Reduce Latency: The Key to Successful Interactive Remote Rendering Systems

Shu Shi

Department of Computer Science

University of Illinois at Urbana-Champaign

Urbana, IL, United States

shushi2@illinois.edu

Abstract—Remote rendering is considered as a general andeffective approach to collaborate mobile devices and cloudcomputing services and deliver computation or network band-width intensive media content (e.g., 3D games, 3D video)to mobile devices. In this abstract, I propose my doctoralthesis research on interaction latency reduction for remoterendering systems. My work mainly focuses on how imagebased rendering techniques can be appropriately applied toreduce the interaction latency which is originally caused bynetwork delay.

Keywords-Remote rendering, interaction latency, 3D video

I. INTRODUCTION

The recent explosion of mobile devices is changing peo-

ple’s computing behaviors and more and more applications

are ported to mobile platforms. However, some applications

(e.g., 3D games, multi-view multi-stream 3D video based

tele-immersive applications [1]) that require intensive com-

putation or network bandwidth are not capable of running

on mobile platforms yet. Remote rendering is considered

as a general and effective approach to deliver computation

or network bandwidth intensive media content to mobile

devices. According to Figure 1, a workstation with enough

computation and network bandwidth resources (e.g., cloud

server) is served as the rendering server. It receives and

renders the source media content (e.g., 3D graphic or 3D

video), and sends the rendering results (2D images) to

one or multiple mobile clients. The mobile client simply

receives and displays the result images. 2D images can be

efficiently compressed with existing video coding tools to

save transmission bandwidth between the rendering server

and mobile clients.

A well known problem of remote rendering is the latency

for the view-change user interaction on mobile clients. If

the mobile user tries to change the rendering viewpoint,

the mobile client needs to send the interaction request

back to the rendering server, and wait for the server to

send back the scene rendered with the updated rendering

viewpoint. Figure 1 shows an illustration of this procedure

with the red dash line. We define the interaction latency

as the time from the generation of interaction request till

the appearance of updated image. Obviously, the interaction

latency takes at least a roundtrip network transmission time

between the mobile client and the rendering server. Given the

Figure 1. Remote Rendering System Framework

unreliable nature of wireless networks, the latency can vary

significantly from time to time and kill the user experience

in many latency-sensitive applications [2].

Therefore, my doctoral thesis research intends to reduce

the interaction latency for remote rendering systems. Ac-

cording to Figure 1, the goal is to reduce the interaction

latency when the mobile client receives a interaction request

that changes the current rendering viewpoint v to v+. If the

mobile client can directly generate the new image I+ (I+

denotes the result image of rendering 3D scene with new

viewpoint v+) in high quality only from the received R

(R denotes the reference frame generated on the rendering

server with old viewpoint v), which is indicated by the

green dash line path, our goal is achieved. Obviously, it

is not enough for the rendering server to simply send the

2D rendering result image. Inspired by previous work in [3]

which suggested using 3D image warping to compensate

latency in remote rendering static 3D models, we apply

image based rendering techniques in our system for dynamic

content rendering. The rendering server is expected to gen-

erate and send auxiliary information (e.g., depth images) as

long as the 2D result image in one reference frame to mobile

clients. The mobile client can use image based rendering

techniques to generate images at the new viewpoint if

any user interaction happens. Compared with static model

rendering in [3], there are more challenges in our research

on rendering dynamic content because all operations should

be finished before the data frame expires. My research will

focus on what reference frames should be selected to support

high quality rendering on mobile side and how reference

frames can be efficiently generated to meet the deadline of

each dynamic data frame.

Fourth Annual PhD Forum on Pervasive Computing and Communications

978-1-4244-9529-0/11/$26.00 ©2011 IEEE 391

My thesis research is novel in two aspects. First, we study

the remote rendering system from a new perspective and

transform the network related interaction latency problem

to a content based reference selection problem. Second, we

try to find a computation efficient method to create an image

based representation for any complex graphic models. The

image based representation is sufficient to support high-

quality rendering of the original model in a limited viewpoint

range. The contribution of my thesis is expected to find

a system solution that integrates image based rendering,

network streaming, cloud computing services, and human

behaviors all together to reduce the interaction latency of

remote rendering systems. In the rest of this abstract, we

discuss the thesis research topics and briefly introduce the

current research status.

II. RESEARCH TOPICS

My thesis research has two general topics. The first topic

is latency reduction. We try to answer four questions:

1) What auxiliary information (reference frame) should

the rendering server generate in order to reduce la-

tency?

2) How does the rendering server generate the reference

frame efficiently?

3) What should the mobile client do if the auxiliary

information is not available?

4) How to scale the system up if one rendering server

needs to serve more than one mobile client?

The second topic is how to evaluate the performance of

latency reduction. In our system, the latency should be

determined by both time and quality of image based ren-

dering. Besides, source content with various motion and

subjective opinions from different individuals can also affect

the evaluation. Thus, we consider the study of latency

reduction evaluation an important topic to understand both

system and human behaviors.

III. STATE OF RESEARCH

We have taken a survey on different previous remote

rendering systems before this thesis proposal. The survey

analyzes the interaction latency of each remote rendering

system design and the details are summarized in [4]. Our

work on 3D video rendering [4][5] answers the question

(1) and (2) as shown above. For each 3D video frame,

two or more depth images are generated on the rendering

server as the reference frame. The mobile client runs the

3D image warping algorithm for every depth image in the

reference frame to generate the image at the target viewpoint

so that the exposure errors caused by single warping are

fixed. Different algorithms on how reference frame should

be selected have been proposed in [4][5]. We also studied

how to use GPU to accelerate search based algorithms in

[6]. For the next stage, we plan to extend our current

Figure 2. Prototype Platform: iPhone Client

research to answer two remaining questions and design more

experiments to better evaluate our system and algorithms.

In addition, we have also built a remote rendering system

prototype (Figure 2) which can render both dynamic 3D

graphics and 3D video for mobile devices. The prototype

provides a good platform to test new ideas on latency

reduction in the future.

IV. ACKNOWLEDGMENTS

I want to thank my adviser Prof. Roy Campbell and co-

adviser Prof. Klara Nahrstedt for their guidance and support.

This research has been supported by the National Science

Foundation under Grant CNS 05-20182 and CNS 07-20702.

REFERENCES

[1] Z. Yang, K. Nahrstedt, Y. Cui, B. Yu, J. Liang, S.-H. Jung, andR. Bajcsy, “Teeve: The next generation architecture for tele-immersive environment,” in ISM. IEEE Computer Society,2005, pp. 112–119.

[2] M. Claypool and K. Claypool, “Latency can kill: precision anddeadline in online games,” in MMSys ’10: Proceedings of thefirst annual ACM SIGMM conference on Multimedia systems.New York, NY, USA: ACM, 2010, pp. 215–222.

[3] W. R. Mark, G. Bishop, and L. McMillan, “Post-renderingimage warping for latency compensation,” Chapel Hill, NC,USA, Tech. Rep., 1996.

[4] S. Shi, M. Kamali, J. C. Hart, K. Nahrstedt, and R. H.Campbell, “A high-quality low-delay remote rendering systemfor 3d video,” in MM ’10: Proceedings of the eighteen ACMinternational conference on Multimedia. New York, NY, USA:ACM, 2010.

[5] S. Shi, W. J. Jeon, K. Nahrstedt, and R. H. Campbell, “Real-time remote rendering of 3d video for mobile devices,” inMM ’09: Proceedings of the seventeen ACM internationalconference on Multimedia. New York, NY, USA: ACM, 2009,pp. 391–400.

[6] W. Yoo, S. Shi, W. J. Jeon, K. Nahrstedt, and R. H. Campbell,“Real-time parallel remote rendering for mobile devices usinggraphics processing units,” in ICME. IEEE, 2010, pp. 902–907.

392