augmented reality system · phone are heads-up display of critical information and a completely...
Post on 11-Jul-2020
2 Views
Preview:
TRANSCRIPT
Augmented Reality System
ECE 4901 -- Senior Design
Spring 2015
Final Report
Team 1510
ECE Project Members:
Daniel Adamowicz
Amy Clark
Victoria Tran
Faculty Advisor
John Chandy
Office: ITE 437
Phone: (860) 486-5047
E-mail: chandy@engr.uconn.edu
Graduate Student Advisor:
Rajeev Ghimire
E-mail: rag10005@engr.uconn.edu
Sponsor:
Qualtech Systems, Inc.
Professor Krishna Pattipati
UConn ECE Department
Summary:
The intent of our project is to create a product that will allow users to efficiently diagnose
and troubleshoot a system through the use of smart glasses. The smart glasses will act as a voice-
controlled user interface for the mobile application PackNGo. The smart glasses will access
Qualtech System’s (QSI) remote diagnostic server (RDS) to retrieve necessary diagnostic and
repair instructions and relay the information back to the user through the smart glasses.
Background:
To diagnose a given system, a technician needs to be familiar with that system.
However, any one technician, no matter how experienced, can not possibly know all the causal
relationships in a system or a family of systems. The problem becomes more acute over time as
newer systems are developed, older systems become less common, and technicians ultimately
have more things to remember. This knowledge gap could be potentially fixed with the use of
diagnostic support systems, whose recommendations can be displayed on devices such as
phones, tablets, or computers. This causes another problem: use of these devices requires the
worker to look away and move their hands from the system, which can negatively impact their
efficiency and safety.
With the use of smart glass technology and Augmented Reality (AR), our group will use
previously developed software to aid workers in diagnostic repair of a wide variety of systems.
The project will aim to improve the accuracy and reduce the chance of worker error due to
inexperience or the need to work with unfamiliar systems. The diagnostic information will be
stored in an XML file and uploaded to the glasses. The application must retrieve information
from the file and display it on the glasses. The main benefits of smart glasses over a mobile
phone are heads-up display of critical information and a completely hands-free experience by
utilizing voice commands.
Solution:
Software:
TEAMS-Remote Diagnostic Server (RDS) is a network-based application that gathers and
processes data taken from active equipment and systems. The resulting diagnostics and system health
information can be accessed via web browser. All results (e.g. procedures, diagnosis, testing) are recorded
to TEAMS-RDS for later analysis and future reference.
PackNGo is a software product that is part of QSI’s remote solutions. It extends the Guided
Troubleshooting functionality from the TEAMS system to smartphones and tablets as a mobile
application. The application utilizes JavaScript and communicates with TEAMS-RDS. When PackNGo is
opened, the user inputs the system being diagnosed and selects any relevant error codes and available
tools from a checklist provided by PackNGo. PackNGo then displays a series of instructional checks
through multiple pages. The user answers yes or no questions to obtain the most likely fault causing the
problem in the system.
One aspect of our design is to take the current UI as shown in Fig. 1 and adapt it to create an
efficient display for smart glass. We plan to utilize this current software and adjust it to be controlled with
voice commands to move through the troubleshooting questions and to bring the user to a repair solution.
The final product should have no remaining touch controls for navigation.
Android Studio is our main source of creating our Glass application. It allows us to create a fully
functional UI and process the information parsing using the Java programming language. Android Studio
is an ideal way to work with a Gradle based system which is what is used for google glass. Android studio
displays the project files in a way that is easy to work with. Each of the classes which are defined as
activities are organized and easy to navigate. The project view in android studio groups the files based on
function. For example it groups the build files, manifest files and resource files into there own folders so
it is easier for us to manage our project. Android studio was the main resource for programming google
glass.
In addition we also used Eclipse to create the parser and check that it was working correctly.
Eclipse can use console commands to output the information and check that it is correct. This is easier
and faster to do than on Android studio. Eclipse and Android studio have the same libraries that we can
use so there is no problem importing the code to Android studio later.
Current UI on mobile device
Fig. 1. Screenshots of current version of PackNGo. The top two screenshots are the lists of user
inputs to customize PackNGo’s instructions. The bottom two screenshots are examples of
PackNGo’s yes/no interface and sample instructions.
Use-Case Diagram
Fig. 2 Use-Case Diagram of network integration of final product
Sequence Diagram
Fig. 3: Sequence diagram, showing ideal flow of interaction of all of the systems in final design
Hardware:
Initially, our team researched technical specifications to narrow down our choice of smart
glasses to two brands: the Epson BT-200 and Google Glass. We considered attributes such as
display size, processing power, battery life, and wifi/bluetooth connectivity. It was also a
requirement that the device have a microphone in order for us to incorporate voice commands.
The Epson BT-200 is a simple smart glasses model we wish to use to integrate PackNGo
with as a proof of concept. The BT-200 is operated by a wired trackpad in the user’s hand which
operates similarly to a touchpad mouse on a laptop computer. The glasses use a 960x540 pixel
display that appears to float in the center of the user’s vision. The main screen is similar to a
desktop on a computer: there are icons for different applications and interaction is done using a
mouse pointer controlled by the touchpad. The relevant technical specs of the BT-200 are 1 GB
of memory, 8 GB of storage space, and Android 4.0.4 operating system. [1]
The battery life for
the Epson is advertised as approximately six hours at room temperature, although our team found
it to be closer to 3-4 hours in testing. Due to the BT-200’s similarity to a normal computer, the
UI changes necessary to integrate the existing PackNGo software is minimal. The BT-200 is
considered only a proof of concept as voice commands are not officially supported and would
require signal processing analysis to fulfill the hands-free requirement of the design.
We have selected Google Glass to be the brand of smart glasses for our final design
solution. Glass supports voice commands, but its general user interface is significantly different
than the BT-200 or a computer. The screen for the glasses is a prism positioned in the top right
corner of the right lens. This leaves the rest of the user’s field of vision unobstructed. Google
Glass has a minimalist style UI that has much less space for text and pictures than a computer
screen. There is also no support for direct mouse clicks or touch control. This means the current
PackNGo design would not work at all on Glass and must be completely redesigned. Glass
operates almost entirely on voice commands by default, and applications written in the Google
Developer Kit (GDK) can easily integrate voice commands. The relevant technical
specifications for Google Glass are 1 GB of memory, 12 GB of storage, and Android 4.4
operating system. [2]
Critics report the battery life of the Glass to be about three to five hours of
extensive use [3]
which we have found to be approximately correct.
Theory:
For the Epson model, the PackNGo functionality is essentially mirrored on the BT-200
interface as the BT-200 is usable similarly to a computer. PackNGo required no additional
modification to function on the BT-200.
For the Glass model, the application must have access to an XML document containing
information for some given system. This XML file is generated by Qualtech software and given
to us directly. The Android application was created from scratch to display the necessary
information in an organized manner. Part of this application parses the XML file and extracts
the necessary information, then displays this information on the Glass. The application is opened
with and fully controlled by voice. After development of this application, we will test and verify
the compatibility of our application with both models of smart glasses. The application running
on the Glass will then be tested in a field environment. The tree diagram can be interpreted as in
Fig. 4:
Fig. 4: Visualization of tree diagram structure
We will need to understand the diagnostic process of PackNGo to ensure that we can
utilize QSI data through the XML file. Contact with QSI chief architect and software engineer
was necessary to avoid complications.
We will need to export the current TEAMS model of air handling unit (AHU) to RDS to
the QSI server to convert diagnostic information to XML file format. This system will be the
test environment for the final product.
The final product should function properly in environments such as automotive facilities,
aerospace industry, hospitals, semiconductor fabrication plants, and more, all of which are
potential QSI clients.
Preliminary Observations:
After some basic testing of the Epson BT-200, we have noticed that the design is not
ideal for a hands free program. First, the touchpad input is connected by a wire. This feature is,
unfortunately, bulky. Ultimately, this means that the BT-200 is not intended to be a hands-free
device without significant manipulation of sensor data within an application. In order to utilize
voice commands on the Epson, third party speech recognition software would need to be
integrated into the QSI software. However, we can still use this device to display PackNGo as is
for a proof of concept for future augmented reality projects.
Another issue is in the way the BT-200 screen is displayed. The screen appears to float
in the center of the user’s field of vision. Although the opacity can be adjusted, it is still possibly
disruptive for someone trying to work on a system. For future augmented reality projects, this
feature could potentially be useful for projecting useful images and data directly onto a system.
For the purposes of this project, the positioning of the screen is only a disadvantage.
The biggest issue with Google Glass is its heat dissipation. We found that using the
Glass for extended periods of time (longer than 30 minutes) could cause the Glass to get
excessively hot. A warning screen appears when reaching critical temperatures. We have
experienced rare system crashes possibly caused by overheating. To remedy this, we
recommend not operating Glass for longer than intervals of thirty minutes, and turning it off in
between intervals to cool. The system could likely operate for longer, but in the interest of being
safe, we recommend limiting its on time to thirty minutes.
Glass voice commands can be overly sensitive for short, simple phrases. When the
microphone detects human voice, it starts attempting to match its microphone data to all possible
voice commands. This means that a voice command could register even if it is said improperly
or incompletely. For example, the command “Take a picture” sometimes works just by the user
saying “Take a pic” or similar incomplete commands. The purpose of this is so that people with
accents or speech impediments can still use the Glass. This causes issues with multiple voice
commands that sound similar and are ready at the same time. A related issue is that once a voice
command is processed (even if the user is not done saying it), the Glass reacts and, depending on
the action, will potentially “ready-up” another set of voice commands.
We encountered the aforementioned problems when using the contextual commands
inside the application. To open the contextual commands, the user must say “ok glass” (this
command can not be changed). After the “ok glass” command is processed, the program then
waits for the next set of commands, which includes the command “Yes”, to continue traversing
the PackNGo system data. We have noticed that:
1) “ok glass” seems to trigger before the phrase is finished,
2) prepares to listen for “Yes”,
3) hears “glass” (or at least, part of the word),
4) and then triggers “Yes” before the user has a chance to provide meaningful input. This
inconsistency in our application could be fixed by changing the voice commands from “Yes” and
“No” to something else; unfortunately, there does not seem to be any logical replacement for
these words that would make sense for any and all given diagnostic prompts. This is simply
because the diagnostic prompts are written in a way so that all PackNGo buttons are the same for
every step (yes and no).
Specifications:
Product Google Glass Epson BT-200
Price ($) $1500.00 $699.99
Display Prism projector, 640×360 pixels 960×540 pixels screen
Onboard Storage (GB) 16 flash total (12 GB of usable
memory)
8 GB
OS Android 4.42 Android 4.0.4
Power Lithium polymer battery (2.1 Wh) Lithium poly 2720mAh
CPU OMAP 4430 SoC dual-core TI OMAP 4460 1.2 GHz dual core
Memory 1GB RAM (682MB available to
developers)
1 GB
Sound Bone conduction transducer headset, surround sound
Camera Photos – 5 MP, videos – 720p VGA
Connectivity Wi-Fi 802.11b/g, Bluetooth, micro
USB
Wi-Fi (IEEE802.11/b/g/n), bluetooth,
microUSB 2.0
Table 1 - Relevant technical information for both brands of smart glasses
Implementation:
The XML parser was written in Java and built using Eclipse. The parser utilized the
DOM and SAX existing parser classes to gather data from the XML. The data we need to
retrieve from the XML is stored in nodes with numbered labels. The data we need to extract
from each node is the label number, instruction string, outcome actions (typically yes/no), and
the outcome pointers. The label number keeps track of which step the program is currently
reading. The instruction string and outcome actions tell the program what to print to the screen.
The outcome pointers tell the program which node to go to next based on what command
(yes/no) the user says. The parser simply scans the XML file for the relevant tag names and
returns the necessary data.
Glass applications can be designed using one of three templates: static cards, live cards,
or immersion. We chose to create an immersion application because it gives us maximum
control over the Glass interface. Immersion applications allow us to continually update
information on the screen without a heavy impact on Glass’s processing power.
Due to Glass’s minimalist UI, the amount of information displayed on each screen is
limited. The yes/no buttons have been changed to voice commands, so they do not take any real
estate on the screen. The pictures and text can be displayed well by resizing them slightly.
Fig. 5: Example of application screen with instruction string.
Fig. 6: Contextual pop-up menu containing typical yes/no choices.
There was one major issue with our original implementation. The XML file to be parsed
was included in the Java package and uploaded to Glass. When the parser runs, the program can
not locate the XML file and is completely unable to retrieve any data. As of this time, the reason
for this problem is unknown. Using Windows Explorer, very limited file paths can be accessed,
so it is difficult or impossible to locate files uploaded to Glass.
As a workaround, we converted the entire XML file to a string and wrote it explicitly
inside the parser code. Doing so requires the XML to be modified to eliminate all quotation
marks (due to Java string syntax), done using a find-and-replace tool in any text editing program.
This string workaround was able to be parsed properly by our application [4]
.
Budget:
The main focus of this project is developing an existing software for use on Glass. The
Epson BT-200 and Google Glass are both provided by the graduate student group and cost our
team nothing. The Integrated Development Environments (IDEs) and all Qualtech software is
free for us to use. Since the entire project is software based, no other parts need to be purchased.
Therefore, the budget for this project is zero.
Timeline:
The main focus of the first semester was to research the many types of different smart
glasses available and choose one most appropriate for the task of aiding remote diagnostics. We
also worked on Android development and gaining access to the current PackNGo software,
TEAMS-RDS. We experimented with BT-200 features and ran the original PackNGo software
on them.
In the second semester, we began full development of the Glass application. A mock-up
UI for the Glass application was designed. We analyzed the XML file and developed the Java
parser. We developed the full Android application with integrated voice commands, and ran the
parser program on the Glass to extract data.
Appendix:
Citations:
[1] http://www.epson.com/cgi-bin/Store/jsp/Product.do?sku=V11H560020
[2] https://support.google.com/glass/answer/3064128?hl=en
[3] http://www.techradar.com/us/reviews/gadgets/google-glass-1152283/review/
Software References:
https://www.teamqsi.com/products/teams-rds/
https://www.teamqsi.com/products/packngo/
http://developer.android.com/tools/studio/index.html
http://eclipse.org/
User Guide:
1) Create an XML file for a system using TEAMS software.
2) The XML file to be used must be modified (refer to Implementation, [4], page 14) and
written directly into the string variable in the Java package.
3) To upload the application, use Android Debug Bridge (ADB) Sideloading. For more
information refer to http://googleglassfans.com/archives/3054/sideload-google-glass-
apps-windows/
4) Once the application is uploaded, open from the main menu with “Ok glass” > “Check
system”
5) The first step of the used XML file will automatically appear. To continue, say “Ok
glass” followed by the related voice commands (e,g, “Yes” or “No”)
6) To go back to the previous step, say “Ok glass” followed by “Go back”
7) To exit the application, swipe down on the Glass touchpad.
Acknowledgements:
We wish to acknowledge and thank the previous senior design teams and graduate
students for providing real-time monitoring and creating a link between the sensor values of the
HVAC system in the Information Technology Engineering Building (ITEB) at UConn, to the
failure protection software, TEAMS, run by QSI. We also wish to thank Qualtech Systems, Inc.
and the University of Connecticut Electrical Engineering Department, for support with funding,
advising, and sponsoring this project. Special thanks to our graduate advisor Rajeev Ghimire
and project advisor John Chandy for guiding us through development.
top related