teleimmersion final report

8/2/2019 Teleimmersion Final Report

http://slidepdf.com/reader/full/teleimmersion-final-report 1/10

Final Report HART Demonstrator Project

Lisa Wymore – Assistant Professor, UC Berkeley Department of Theater, Dance,

and Performance Studies

The project is entitled:

Portable Durable Tele-Immersion Technology for Teaching, Artistic Performance,

Global Networking, and Experiments in Creativity and Collaboration

The project was initially conceived in Summer 2008. At that time I was creating a large-

scale performance project in conjunction with the Merce Cunningham Dance Companyresidency at UC Berkeley for a November 2008 performance. Cal Performances, the

prestigious presenting organization at UC Berkeley, produced the event on November

14th. The perfor mance was entitled: Panorama-Multi-Media

Happening

and involved

numerous art and technology units across campus. For Panoram

a

I continued my creative

explorations with Tele-Immersion, a technology that I have been working with since

2005 with the labs here at UC Berkeley and at the Univer sity of Illinois predominately.

For this project it became clear that we needed to create a portable Tele-Immersion

system for the November 14 performance so that Tele-Immersion could be experiencedt h

by the audience members attending the event. So, working with Ruzena Bajcsy’s Tele-

Immersion lab within the Electrical Engineer ing and Computer Science Department at

UC Berkeley, we cr eated two portable systems that existed 100 feet apart from each

other. User s could engage with the system and see one another across the performance

space and also on large monitors sharing the virtual “meeting environment”.

For those unfamiliar with Tele-Immersion technology, it can best be described as adistributed collaboration tool that utilizes 3D image based rendering of users in real time.

Remote users can meet in virtual envir onments and share virtual worlds together, not as

traditional avatars which use animation graphics, but as new kinds of avatars that use real

time motion and video based images of users. Further more, any 3D data can be uploadedto the virtual environment to be investigated within collaborative virtual settings.

Examples of the kind of work we do in the UC Berkeley Tele- Immersion lab can be seenat the following website:

http://tele-immersion.citris-uc.org/video

This HART Demonstr ation Project funded a graduate student, Ram Vasudevan, to create

a more robust ver sion of the portable Tele-Immersion system to be used within a

dedicated dance studio space within my Department of Theater, Dance, and Performance

Studies. The larger goal of this project is to create what I am calling Integrated Distance

Labs ( ID-labs) within other art/dance based departments across the country and hopefullythroughout the world, that allows a new kind of choreography to emerge. We are also

working on the integration of body based sensors and sound within the system. We have

tests with UC Merced and UC Davis planned this coming spring 2010, as well as tests on

the UC campus between the dance studio “I D-lab” and the Tele-Immer sion lab in

Electrical Engineering and Computer Science. Furthermore, in spr ing 2011 an entireevening of perfor mance will be dedicated to questions around dance and technology and



the portable Tele-I mmersion lab, that HART has funded, will be at the heart of this

perfor mance.

I mention these things because they address the major issues that we are tackling with

Tele-I mmersion at this time: 1. The robustness and transferability of Tele-I mmersion

technology to a larger public, and 2. The integration of sound and body based sensors

into Tele-Immersion technology, in real time, to enhance remote user experience andcollaboration within the system. The hope is that the expansion of Tele-Immersion labs,

using the same vision and rendering protocols will allow for a network of distributed labs

to communicate and collaborate around a variety of 3D data and information.

The results that we have produced from our Demonstration Project funding:

1. A highly functional lab has been built in Zellerbach Room 170 dance studio. This

space is now a dedicated “dance lab” and will soon become an Integrated Dance

Lab (ID- lab) with body based sensors, surround sound speakers and dedicated

Tele-I mmersion technology installed into the space permanently

2. All equipment needed for the Portable Tele-Immersion system has been

purchased and tested.

a. Projection equipment is installed

b. Special stage lights have installed

c. Internet has been installed (3 IP addresses)

d. Adequate electricity has been installed

e. Proper projection sur faces and flooring has been installed

3. The renderer necessary for bi-located Tele-Immersion has been updated in

collaboration with UC Davis’s CAVE labs.

4. Intense software updates and code updates have been completed by Ram

Vasudevan (funded by the HART Demonstration project) and the numer ous other

computer scientists and engineers working in the UC Tele-Immersion lab.5. A test between the two UC Berkeley Tele- Immersion labs (dance studio and

EECS lab) has occurred using one of the 3D cameras. (see images below) Later

this f all we will test with four 3D cameras.

6. Future tests are scheduled between UC Merced and UC Davis with regard to

Tele-I mmersion technology in general. See website listed above for videos

showing the kinds of collaboration that we are working on.7. Other Dance Programs and Departments are showing interest in developing Tele-

Immersion (ID-labs). Namely UC Santa Cruz and UC Irvine will be working in

partnership with the UC Berkeley labs over the next two years.



Regarding my academic efforts: I t is hard for me to repor t on this in the tr aditional sense.

I am a chor eographer working with electrical engineers and computer scientists on ahighly complex and multi-modal project within a research based institution. The only

way that I can answer this is to say that the Demonstration Project is inspiring on many

levels to me and in a sense inspiration and creativity are my resear ch. Dance involves

bodies in space and time and Tele-Immersion technology prioritizes bodies in space andtime. It also involves people collaborating together using visual feedback, which is

another means by which dance choreography is made. So in essence, by work with Tele-

Immersion in general has been extremely helpful to my research as a dancechoreographer. With regard to the HART demonstration project, bringing a portable

Tele-I mmersion lab into a dance studio is unprecedented.

It is my hope that by creating this lab it will become a beacon for other departments to do

the same so that a new kind of body based interactive mechanism for collaboration and

communication can be developed. This tactic has been working. As mentioned above,

other UC dance departments ar e getting involved and a renowned dance and technology

artist from the University of Utah, Ellen Bromberg, will be on campus in spring 2011 towork directly with the dance studio Portable Tele-Immersion lab, funded by HART. So

this pr oject has had incredible impact on my visibility as a dance ar tist who engages with

and collaborates with technology. This has had a positive effect f or the public perception

of UC Berkeley’s engagement with new technologies in relationship to the arts in

particular.

With regard to impact within the larger Project Bamboo community, it is difficult to say.Project Bamboo prioritizes easily shared programs and established digital systems rather

than development of physical labs. It is my hope that Tele- Immersion technology remains

one of the future concepts for Pr oject Bamboo to invest in. One can imagine that a

collaborative 3D environment in real time, based on real body movements and real bodyimages will one day be highly usef ul and “gr een” technology. It goes beyond traditional

webcasts or tele-conferences because information can be viewed from infinite angles, the

user can engage with the system as either first person or viewer, there is infinite amountof digital data that can be used within the system for analysis, collaboration, and

archiving purposes. In some ways it is a kind of futuristic library or public

space/commons – a place to come and find information, meet in virtual rooms or

environments, make things in collaboration, explore through multi-modal approaches

(vision, sound, body based sensors, robotic interaction with data, etc.)



5. How do we tackle issues of internet bandwidth for real time interactivity.

6. Is off-line data useful to other humanists? And should more of f-line engagement

with Tele-Immersion be investigated?

Regarding lessons learned from this project:

As I write this section I am realizing that I am continually learning from this project. Themajor concepts and ideas that I take away from this project are:

1. It is essential to have engaged and talented graduate students working on multi-

disciplinary projects. I cannot speak highly enough of Ram Vasudevan, who was

the graduate student funded thr ough this Demonstration Project.

2. Collaboration on a pr oject such as this requires weekly meetings and involvement

from all who work in the lab; even if other projects seem unrelated the exchangeof infor mation and ideas that comes f rom meeting is essential.

3. Technology labs are filled with extremely creative people who often don’t always

speak the same language. And being an artist entering this world it requires

patience to become an active participant. Interaction over a long period of time,

sharing of presentations and regular meetings are essential for success.4. It takes much longer to accomplish a specific goal within a collaborative setting.

Often the engineer s that I work with ar e engaged with other research projects and

it takes time and perseverance to create a successful collaboration between artists

and technologists.

From the technologist perspective Ram Vasudevan, has learned much from his work on

this Demonstration Project. He has established an entirely novel way of dealing with

images and texture within the Tele-I mmersion system. All together there are 48 cameras,which can simultaneously capture images with 15 – 20 frames per second. The clusters

are arranged around the user to capture a full-body 360-degree model in real time. Each

of the clusters performs stereo calculation on pairs of cameras by matching reliablefeatures. The features are selected using a novel approach by first performing

triangulation of the image, which divides the image into triangular segments of various

sizes depending on the texture and color consistency of the scene. The homogenous (i.e.

textureless) regions result in larger triangles while the textured regions have many small

triangles. The stereo matching is performed on the acquired triangle nodes.Using this tr iangulation has several benefits, (1) it helps find reliable features, (2) it

reduces the number of matches that need to be performed, (3) it keeps a global structure

of the 3D representation, and (4) it allows for efficient compr ession of the data. This

adaptive triangulation allows for fast (under 25 ms) and robust real-time 3D

reconstruction. The reconstruction rate is, however, limited by the image acquisition rate

of the cameras and the networ k bandwidth to the frame rate of about 20 FPS.

Once the stereo calculation is performed, the triangulation is encoded along with thecolor and dispar ity information for each node. The data is packaged and sent to the local

renderer or remote gateway. The gateway controls the network traffic by routing the

packets over long distances to minimize the delays. Once the renderer receives the data,

3D mesh is extracted and transformed into a common coordinate system. The renderer



needs to have exact information of the camer a clusters, which is obtained by prior

geometric camera calibration. The render ing algorithm combines the information from

diff erent views by weighted contribution from different clusters, (overlapping clustersmay reconstruct the same surface) using real-time ray-tracing technique to generate

appropriate images in the visual buffer.

Future Directions:As one can see from the above information, Tele-Immersion technology is a complex

system involving endless calibration, software and hardware updates, and

synchronization of data from distributed labs. The Demonstrator Project f unding has beenessential in establishing a kind of codified protocol for futur e work in this area. Ram’s

mathematical calculations, attention to calibration and his “cleaning up” of the code have

made for a more robust and realistic experience. Gregorij Kurillo, the Head Engineer in

the lab, has also added to the system incredibly by establishing renderer protocols,

expanding the visual perspective within the system, adding new 3D data, establishing

collaboration with UC Merced and UC Davis, and much mor e than I can list here. Also,

post-docs, grad students and visiting computer scientist from all over the world have

contributed to the lab in areas r elated to visual texture, human motion detection,skeletonization, sensors, robotics, and the like. It is a VERY active lab, led by Ruzena

Bajscy.

I have ever y intention of continuing my work as collaborator in Tele-Immersion

technology – adding my expertise with regard to body movement, collaboration

protocols, creativity enhancement, live performance engagement, and improvement of

computer human interaction. Another aspect of the pr oject that I would like to seek support for relates more to project Bamboo and that is the creation of an inter active web-

site (or shared tool) where protocols, programs, lab designs, best practices, equipment

lists, data sets, results from experiments, publications, ongoing discourses, etc. are made

public. The desire is to foster this kind of open engagement with the public so that Tele-Immersion technology is adopted within other university/research settings and

infor mation is shared.

As I mentioned earlier in this report, my future work will involve the creation of an

Integrated Distance Lab (I D-lab). I am currently working with Po Yan, and engineer

within the UC Berkeley Tele-Immersion lab, to work with body based sensors and sound.

Imagine a moment when a dancer enters a 3D Tele-Immersive environment surrounded

by multiple 3D digital cameras and displays, where internal and external cues for creative

movements come not only from physical objects located within the 3D Tele-Immersive

virtual space but also from remote participants who are geographically distributed, yetvisually present in the virtual space. The dancers collaborate with one another utilizing

visual cues to solve choreographic problems or to investigate new movement potential

within this virtual dance studio. Suddenly the choreographer has an entir ely new

paradigm within which to make creative choices. Now imagine adding a somatically

derived oper ation to this model – the addition of body-based sensors whose outputs can be synchronized to sound, and this sound can be streamed, in near real time, and with



some synchronized accuracy, to the visual data being received by the user . The 3D Tele-

Immersion system will now allow user s to not only see one another but also “hear” each

other. In a way it is like they can sense each other’s bodily responses through the digitaldivide and make movement decisions based on this information.

The sensors can be synchronized to realistic body sounds ie: the heartbeat, or they can be

linked to musical digital interfaces that allow for infinite sonic possibilities - using avisual programming language f or multi-media called Max/MSP1. The dancer s will then

be able to perceive each other through both visual an

d

audio feedback within the virtual

dance studio. Offline, and in the "real! dance studio, the main way dancers sense eachother is through listening to breath patterning and through the sensing of other dancers’

levels of exertion and effort. The ID-lab seeks to replicate, albeit through a different

mode of awareness, the feelings that dancers experience within "real” dance studios. The

ID-lab will do this by attaching small, very sensitive, microphones near the user’s mouth

and also by adding two sensitive body sensors: a physiological biosensor that captures

heart rate and br eath patterns; and an accelerometer that captures velocity of gesture and

spatial intent of gesture.

Using MaxMSP the output from these sensors will be converted to sound files locally and

streamed to the participating Integrated Distance Labs. The user at each site will hear the

streamed sound files from the participating Integrated Distance Lab via a surround sound

speaker system. Each lab will be equipped with the same sound system for calibration,

and synchronization purposes. With the sound being experienced "around! the user and

the sounds being heard constructed from the users’ bodies themselves, a r ich audio wor ld

will be created. A computer scientist and signal-processing expert, Gerald Friedland, isadvising the pr oject on the audio and sensor data streaming parts.

The Demonstrator Process: Should demonstrators be included as part of future Bambooactivity? I f so, how would you improve the demonstrator process? I have just a few

suggestions:

1. I felt that the day the UC Berkeley HART funded projects wer e able to sharedisplays with one another was really useful. It was a kind of mini-conference

where ideas were shared, people got to mingle and look at displays/tables, and

talk about their projects with on another. I wonder how that collegially experience

could continue? Per haps another mini- conference or open studio to visit each

other’s work areas?

2. I wonder if Demonstration Projects fr om other campuses could post slides or

mini-presentations on the Bamboo site? Perhaps they do already and I am missing

this? But it relates in some ways to the “story collection” or “recipe gathering”that Project Bamboo is wor king into their system.

3. I would love to be able to demonstrate the Portable Tele-Immersion system

sometime at a Project Bamboo conference, and see other practice/site/lab-based

technologies as well demonstrated live.

4. I miss seeing a kind of live, body based interaction with technology within theBamboo community, but this is my bias. I realize that this type of wor k is not



necessary considered “shared”. That is can we interact with technologies at

Bamboo conferences more? We sometimes have web casts but what else can we

“work with” live in real time?

Thank you so much for funding my wor k as a Demonstration Project. I look f orward

to being part of the Bamboo Community for a long time to come!

Lisa Wymor e

teleimmersion final report

Documents