Demo: Practical Mobile Augmented RealityPuneet Jain

Duke UniversityJustin Manweiler

IBM ResearchRomit Roy Choudhury

UIUC

ABSTRACTWe present demonstration of real-time indoor mobile aug-mented reality (MAR) on off-the-shelf smartphones. Ourproposed system OverLay allows random physical objecttagging from camera’s viewfinder. Later, these tags can beseen by others from different angles, locations, and times.Our approach does not require changes to infrastructure,localization schemes, specialized cameras, or modificationto phone’s operating system. Designed and developed forcurrent generation smartphones, our experiments in an in-door setup of 100 objects and 10 volunteers, OverLay showspromising 95% precision, 85% recall, and 480ms end-to-endlatency in the median case. If made commercially available,OverLay can immediately apply to city tourism, PoI discov-ery, infrastructure maintenance, and object privacy.

Demonstration SetupWe expect to demonstrate our live system on Android smart-phones in HotMobile 2015 workshop. The users shall beable to experience OverLay on provided smartphones (op-tion of installing on attendees personal devices is under-consideration). The participants shall be able to tag orretrieve (subjected already tagged) random objects in theworkshop venue by pointing camera at them. We shall al-low simultaneous tagging and retrieval from multiple usersto demonstrate that same object can be retrieved from dif-ferent locations, angles, and phones. Moreover, a videodemonstration of OverLay featuring first hand experienceof a volunteer inside Coordinated Science Lab at UIUC willbe shown during the demo session.

Learning Module

Annotation DB

GPU Optimized Pipeline

SURF RefineMatch

(time, orientation)

frame

“Sculpture:$55”

Select Candidates

Geometric Representation

Learning Module

Blur?

Hand Motion?

(frames, sensors)

Annotate

(image, “Sculpture: $55”)

NETWORK

SURF

Selected candidates

Update m

odules

Retrieve

features/descriptors

Figure 1: Schematic diagram of OverLay systemshowing tagging and retrieval process.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s).HotMobile’15, February 12–13, 2015, Santa Fe, New Mexico, USA.ACM 978-1-4503-2793-0/14/06.http://dx.doi.org/10.1145/2594368.2601472 .

System DetailsReal-time MAR is a challenging problem because of trade-offs between accuracy, desired user experience, and practicaldeployment assumptions. Some of the well known challengesare as follows:

1. MAR requires indoor localization to reduce candidatesearch space. Unfortunately, best known practicalschemes perform in the order of meters. Similarly,precise attitude estimation is also not possible becauseof noisy onboard sensors.

2. MAR requires heavyweight computer vision to run ondevice but current generation smartphones have lim-ited processing power.

3. MAR may be enabled by offloading computation to thecloud but end-to-end network latency is too high forper frame offloading resulting unacceptable end-userexperience.

Our exhaustive experimentation with vision and sensingbased approaches in isolation conclude that neither of thetwo can solve this problem in real-time with acceptable accu-racy. The vision based approaches require matching presentcamera view with existing set of tags in the database. Dueto sheer computation requirement on a practical databasesize, this operation is not possible in real-time. Opportuni-ties needs to be identified in leveraging lightweight sensors toreduce the search database size before applying heavyweightvision algorithms for the accurate matching.

Figure 1 shows architecture diagram of OverLay. Our coreresearch contribution in the design of OverLay is in building“location-free” geometric model of the surroundings usinginertial sensors. This model consists of relative position-ing of objects in space and time — learnt from the naturaluse of our application. We utilize this geometric model anda past series of retrievals to reduce the candidate tags forfuture retrievals. Putting together this optimization on aGPU based pipeline, we are able to reduce processing la-tency from tens of seconds to 480ms without compromisingon the accuracy. OverLay client is an Android app runningon Samsung Galaxy S4 phones. OverLay server is a remoteGPU equipped desktop computer satisfying our data stor-age and computation offloading requirements. An exampleretrieval from OverLay application is shown in Figure 2.

Figure 2: Live object retrieval using OverLay.