sixth sense technology 04

SIXTH SENSE TECHNOLOGY

1. INTRODUCTION

Sixth Sense Technology is a mini-projector coupled with a camera and a cell phone

which acts as the computer and connected to the Cloud, all the information stored on

the web. Sixth Sense can also obey hand gestures. The camera recognizes objects

around a person instantly, with the micro-projector overlaying the information on any

surface, including the object itself or hand. Also can access or manipulate the

information using fingers.

Sixth Sense is a wearable gestural interface that augments the physical world around

us with digital information and lets us use natural hand gestures to interact with that

information. By using a camera and a tiny projector mounted in a pendant like

wearable device. Sixth Sense sees what you see and visually augments any surfaces or

objects we are interacting with. It projects information onto surfaces, walls, and

physical objects around us, and lets us interact with the projected information through

natural hand gestures, arm movements, or our interaction with the object itself. Sixth

Sense attempts to free information from its confines by seamlessly integrating it with

reality , and thus making the entire world your computer.[1]

Fig 1. Parts used in Sixth Sense Technology [1]

Akshay .N .Mahajan(120573116004)


2. HOW SIXTH SENSE TECHNOLOGY WORKS

The technology in itself is nothing more than the combination of some

stunning technologies, but the idea of combining those technologies is really

great.

The technology is mainly based on hand gesture recognition, image capturing,

processing, and manipulation, etc.

The camera is used to recognize and track user’s hand gestures and physical

objects using computer-vision based techniques, while the projector is used to

project visual information on walls or on any physical thing around us. Other

hardware includes mirror and colored caps to be used for fingers

The projector: projects visual information, enabling surfaces, walls and

physical objects around the wearer to be used as interfaces;

The camera and hands: recognizes and tracks the user's hand gestures and

physical objects using computer-vision based techniques.

The software program: processes the video stream data captured by the camera

and tracks the locations of the colored markers (visual tracking of colored

fingers) at the tip of the user’s fingers using simple computer-vision

techniques. The movements and arrangements of these fiducials are

interpreted into gestures that act as interaction instructions for the projected

application interfaces.

The maximum number of tracked fingers is only constrained by the number of

unique fiducials, thus Sixth Sense also supports multi-touch and multi-user

interaction.

3. ARCHITECTURE OF SIXTH SENSE TECHNOLOGY [3]



Fig 2: Architecture of Sixth Sense Technology[3]

Sixth Sense, a tool that connects the physical world with the world of data.

Digital prototype called the “Sixth Sense”, which wearable hi-tech devices

that can turn anything into a touch screen. Your wrist can become a watch and

your palm a phone. The device is combination of a wearable camera, a

projector and a mirror where the camera recognizes movements of the fingers

of your hand and translates those into commands.

The hardware components are coupled in a pendant like wearable device.

Both the projector and the camera are connected to the mobile computing

device in the user’s pocket. The projector projects visual information enabling

surfaces, walls and physical objects around us to be used as interfaces; while

the camera recognizes and tracks user’s hand gestures and physical objects

using computer vision based techniques. The software program processes the

video stream data captured by the camera and tracks the locations of the

colored markers (visual tracking fiducials) at the tip of the user’s fingers using

simple computer vision techniques.



The movements and arrangements of these fiducials are interpreted into

gestures that act as interaction instructions for the projected application

interfaces. The maximum number of tracked fingers is only constrained by the

number of unique fiducials, thus Sixth Sense also supports multi touch and

multi user interaction.

3.1 POCKET PROJECTOR [3]

Pocket projector is an emerging technology that applies the use of a projector in a

handheld device. It is also know as Handheld projector, pocket projector or mobile

projector or Pico projector. It is a response to the emergence of compact portable

devices such as mobile phones, personal digital assistants and digital cameras. Which

have sufficient storage capacity to handle presentation materials but little space to

accommodate an attached display screen. Handheld projectors involve miniaturized

hardware and software that can project digital images onto any nearby viewing

surface, such as a wall. This entire system is compacted into one very tiny chip. An

important design characteristic of a handheld projector is the ability to project a clear

image, regardless of the physical characteristics of the viewing surface.

3.2 PARTS OF POCKET PROJECTOR [3]

The system comprises four main parts:

The electronics

The laser light sources

The combiner optic, and the

Scanning mirrors.

3.3 THE WORKING OF SIXTH SENSE DEVICE [4]



Fig.3 Working Of Sixth Sense [4]

First, the electronics system turns the image into an electronic signal.

Next the electronic signals drive laser light sources with different colors and

intensities down different paths.

In the combiner optic the different light paths are combined into one path

demonstrating a pallet of colors.

Finally, the mirrors copy the image pixel by pixel and can then project the

image.

3.4 APPLICATIONS [8]

Handheld projector can be used for similar applications than small conventional

projectors. Study applications that are specifically designed for handheld projectors

often using prototypes of mobile phones with an integrated projector.

Gaming: Handheld projectors, in particular projector phones, could offer new

possibilities for mobile gaming as demonstrated by the adaptation of the

Playstation3 game Little Big Planet.



Hand gesture recognition: Size reduction of mobile devices is often limited

by the size of the used display. Apart from the display a complete phone can

be, for example, integrated in a headset. It has been demonstrated that Pico

projectors integrated in headsets could be used as interaction devices e.g.

using additional hand and finger tracking.

Picture browsing: Projector phones allow photographs to be shared with a

larger audience. The projected display allowed viewing of pictures by all

present which is currently not possible using a single mobile device. [4]

3.5 MOBILE COMPUTING DEVICES

A mobile device (also known as cell phone device, handheld device, handheld

computer, "Palmtop" or simply handheld) is a pocket-sized computing device,

typically having a display screen with touch input or a miniature keyboard. In the case

of the personal digital assistant (PDA) the input and output are combined into a touch-

screen interface. Smart phones and PDAs are popular amongst those who require the

assistance and convenience of a conventional computer, in environments where

carrying one would not be practical.[4]

3.5.1 TYPES OF MOBILE DEVICES [4]

Mobile devices have been designed for many applications and include:

Mobile computers.

Graphing calculator.

Handheld game consoles.

Digital still camera (DSC).

Digital video camera (DVC or digital camcorder).

Portable media player.

E-book reader.

Mobile phone.

Personal navigation devices(PNDs)



3.5.2 USAGE

Handheld devices have become use in mobile field management situations to record

information in the field. They are used to achieve a variety of tasks for increasing

efficiency that include digitizing notes , sending and receiving invoices, assets

management, recording signatures, managing parts and scanning barcodes. Handheld

computers used at work have molded over time into a variety of form including smart

phones on the low end, handheld PDAs, Ultra Mobile PCs, tablet PCs .[4]

4. WEARABLE GESTURAL INTERFACE [5]

Information is traditionally confined to paper or digitally to a screen. In this paper,

we introduce WUW, a wearable gestural interface, which attempts to bring

information out into the tangible world. By using a tiny projector and a camera

mounted on a hat or coupled in a pendant like wearable device, WUW sees what the

user sees and visually augments surfaces or physical objects the user is interacting

with.WUW projects information onto surfaces, walls, and physical objects around us,

and lets the user interact with the projected information through natural hand gestures,

arm movements, or interaction with the object itself.

A wearable device with a projector that allows to augment reality with a pendant

projector. The device (via the web cam and coded finger-gloves) recognizes users

gestures while searching for information, taking photos, making a call and presents it

back to the user projected at any display.

The recent advent of novel sensing and display technologies has encouraged the

development of a variety of multi-touch and gesture based interactive systems. Such

systems have moved computing onto surfaces such as tables and walls and brought

the input and projection of digital interfaces into one-to-one correspondence such that

the user may interact directly with information using touch and natural hand gestures.

A parallel trend, the miniaturization of mobile computing devices permit “anywhere”

access to information, so we are always connected to the digital world. Unfortunately,

most gestural and multi-touch based interactive systems are not mobile and small



mobile devices fail to provide the intuitive experience of full-sized gestural systems.

Moreover, information still resides on screens or dedicated projection surfaces. There

is no link between our interaction with these digital devices and interaction with the

physical world around us.

While it is impractical to modify all physical objects and surfaces into interactive

touch-screens, and to turn everything into network enabled devices, it is possible to

augment environment around us with visual information. When digital information is

projected onto physical objects, instinctively it makes sense to interact with it in

similar patterns as with physical objects: through hand gestures.

In this paper, we present WUW, a computer-vision based wearable and gestural

information interface that augments the physical world around us with digital

information and proposes natural hand gestures as the mechanism to interact with that

information. It is not the aim of this research to present an alternative to multi-touch

or gesture recognition technology, but rather to explore the novel free-hand gestural

interaction that WUW proposes.

4.1 What is WUW? [5]

WUW is a wearable gestural interface. It consists of a camera and a small projector

mounted on a hat or coupled in a pendant like mobile wearable device. The camera

sees what the user sees and the projector visually augments surfaces or physical

objects that the user is interacting with. WUW projects information onto the surfaces,

walls, and physical objects around the user, and lets the user interact with the

projected information through natural hand gestures, arm movements, or direct

manipulation of the object itself.

The tiny projector is connected to a laptop or mobile device and projects visual

information enabling surfaces, walls and physical objects around us to be used as

interfaces; while the camera tracks user hand gestures using simple computer vision

based techniques. The current WUW prototype implements several applications that

demonstrate the usefulness, viability and flexibility of the system. The map

application(see Fig.3) lets the user navigate a map displayed on a nearby surface



using familiar hand gestures, letting the user zoom in, zoom out or pan using intuitive

hand movements. The drawing application (see Fig.4) lets the user draw on any

surface by tracking the fingertip movements of the user’s index finger.

The WUW prototype also implements a gestural camera that takes photos of the scene

the user is looking at by detecting the ‘framing’ gesture (see Fig.5). The user can stop

by any surface or wall and flick through the photos he/she has taken. The WUW

system also augments physical objects the user is interacting with by projecting more

information about these objects projected on them. For example, a newspaper can

show live video news or dynamic information can be provided on a regular piece of

paper (see Fig.6).The gesture of drawing a circle on the user’s wrist projects an analog

watch (see Fig.4). The WUW system also supports multiple users collaborating and

interacting with the projected digital information using their hand movements and

gestures. [5]

Fig.4: Map Application [5]



Fig.5: Drawing Application [5]

Fig.6: Framing Gesture [19]

Fig.7: Newspaper [5]



Fig.8:Analog watch [5]

4.2 WUW prototype [5]

The WUW prototype is comprised of three main hardware components: a pocket

projector (3MMPro110), a camera (Logitech Quick Cam) and a laptop computer. The

software for the WUW prototype is developed on a Microsoft Windows platform

using C#. Both the projector and the camera are mounted on a hat and connected to

the laptop computer in the user’s back pack. The prototype system uses plastic

colored-markers (e.g. caps of whiteboard markers) as the visual tracking fiducials.

Fig.9: WUW prototype system[5]



The movements and arrangements of these color markers are interpreted into gestures

that act as interaction mechanism for the application interfaces projected by the

projector. The current prototype only tracks 4 fingers – index fingers and thumbs.

This is largely due to the relative importance of the index finger and thumb in natural

hand gestures, tracking these WUW is able to recognize a wide variety of gestures

and postures. The maximum number of tracked fingers is only constrained by the

number of unique fiducials, thus the WUW also supports multi-user interaction.

The prototype system also implements a few object augmentation applications where

the camera recognizes and tracks physical objects such as a newspaper, a coffee cup,

or a book and the projector projects relevant visual information on them. WUW uses

computer-vision techniques in order to track objects and align projected visual

information by matching pre-printed static color markers (or patterns) on these objects

with the markers projected by the projector.

4.3 CHALLANGES

There are many challenges associated with the accuracy and usefulness of gesture

recognition software. For image-based gesture recognition there are limitations on the

equipment used and image noise. Images or video may not be under consistent

lighting, or in the same location. Items in the background or distinct features of the

users may make recognition more difficult. The variety of implementations for image-

based gesture recognition may also cause issue for viability of the technology to

general usage. For example, an algorithm calibrated for one camera may not work for

a different camera. The amount of background noise also causes tracking and

recognition difficulties, especially when occlusions (partial and full) occur.

Furthermore, the distance from the camera, and the camera's resolution and quality,

also cause variations in recognition accuracy. [5]

5. HAND GESTURE RECOGNITION [6]

Hand gesture recognition is a topic in computer science and language technology with

the goal of interpreting human gestures via mathematical algorithms. Gestures can


http://en.wikipedia.org/wiki/Language_technology


originate from any bodily motion or state but commonly originate from the face or

hand. Current focuses in the field include emotion recognition from the face and hand

gesture recognition. Many approaches have been made using cameras and computer

vision algorithms to interpret sign language. However, the identification and

recognition of posture, gait, proxemics, and human behaviors is also the subject of

gesture recognition techniques.

Hand gesture recognition can be seen as a way for computers to begin to understand

human body language, thus building a richer bridge between machines and humans

than primitive text user interfaces or even GUIs (graphical user interfaces), which still

limit the majority of input to keyboard and mouse.

Hand gesture recognition enables humans to interface with the machine (HMI) and

interact naturally without any mechanical devices. Using the concept of gesture

recognition, it is possible to point a finger at the computer screen so that the cursor

will move accordingly. This could potentially make conventional input devices such

as mouse, keyboards and even touch-screen redundant. It can be conducted with

techniques from computer vision and image processing. The literature includes

ongoing work in the computer vision field on capturing gestures or more general

human pose and movements by cameras connected to computer .Gesture recognition

can be conducted with techniques from computer vision and image processing. The

literature includes ongoing work in the computer vision field on capturing gestures or

more general human pose and movements by cameras connected to a computer.


http://en.wikipedia.org/wiki/Cursor_(computers)

http://en.wikipedia.org/wiki/Computer_screen

http://en.wikipedia.org/wiki/Proxemics


Fig.10: A child being sensed by a simple hand gesture recognition

detecting hand location and movement.[6]

5.1 USES [6]

Gesture recognition is useful for processing information from humans which is not

conveyed through speech or type. As well, there are various types of gestures which

can be identified by computers.

Sign language recognition: - Just as speech recognition can transcribe speech

to text, certain types of gesture recognition software can transcribe the

symbols represented through sign language into text.

For socially assistive robotics: - By using proper sensors (accelerometers and

gyros) worn on the body of a patient and by reading the values from those

sensors, robots can assist in patient rehabilitation. The best example can be

stroke rehabilitation.

Directional indication through pointing: - Pointing has a very specific

purpose in our society, to reference an object or location based on its position

relative to ourselves. The use of gesture recognition to determine where a

person is pointing is useful for identifying the context of statements or

instructions. This application is of particular interest in the field of robotics.

Control through facial gestures: - Controlling a computer through facial

gestures is a useful application of gesture recognition for users who may not

physically be able to use a mouse or keyboard. Eye tracking in particular may

be of use for controlling cursor motion or focusing on elements of a display.

Alternative computer interfaces: - Foregoing the traditional keyboard and

mouse setup to interact with a computer, strong gesture recognition could

allow users to accomplish frequent or common tasks using hand or face

gestures to a camera.

Immersive game technology: - Gestures can be used to control interactions

within video games to try and make the game player's experience more

interactive or immersive.

Virtual controllers: - For systems where the act of finding or acquiring a

physical controller could require too much time, gestures can be used as an



alternative control mechanism. Controlling secondary devices in a car or

controlling a television set are examples of such usage.

Affective computing: - In affective computing, gesture recognition is used in

the process of identifying emotional expression through computer systems.

Remote control: - Through the use of gesture recognition, remote control with

the wave of a hand of various devices is possible. The signal must not only

indicate the desired response, but also which device to be controlled.

5.2 INPUT DEVICES

The ability to track a person's movements and determine what gestures they may be

performing can be achieved through various tools. Although there is a large amount of

research done in image/video based gesture recognition, there is some variation within

the tools and environments used between implementations.

Depth-aware cameras: - Using specialized cameras such as time-of-flight

cameras, one can generate a depth map of what is being seen through the

camera at a short range, and use this data to approximate a 3d representation

of what is being seen. These can be effective for detection of hand gestures

due to their short range capabilities. [6]

Stereo cameras: - Using two cameras whose relations to one another are

known, a 3d representation can be approximated by the output of the cameras.

To get the cameras' relations, one can use a positioning reference such as a

lexian-stripe or in framed emitters. [6]

Controller-based gestures: - These controllers act as an extension of the

body so that when gestures are performed, some of their motion can be

conveniently captured by software. Mouse gestures are one such example,

where the motion of the mouse is correlated to a symbol being drawn by a

person's hand, as is the Wii Remote, which can study changes in acceleration

over time to represent gestures.[6]

Single camera: - A normal camera can be used for gesture recognition where

the resources/environment would not be convenient for other forms of image-

based recognition. Although not necessarily as effective as stereo or depth

aware cameras, using a single camera allows a greater possibility of

accessibility to a wider audience.[6]



5.3 MULTI TOUCH [10]

Multi-touch is an enhancement to touch screen technology, which provides the user

with the ability to apply multiple finger gestures simultaneously onto the electronic

visual display to send complex commands to the device. The term multi-touch is a

trade mark of Apple Inc.

Multi-touch has been implemented in several different ways, depending on the size

and type of interface. Both touch tables and touch walls project an image through

acrylic or glass, and then backlight the image with LED's. When a finger or an object

touches the surface, causing the light to scatter, the reflection is caught with sensors or

cameras that send the data to software which dictates response to the touch, depending

on the type of reflection measured. Touch surfaces can also be made pressure-

sensitive by the addition of a pressure-sensitive coating that flexes differently

depending on how firmly it is pressed, altering the reflection. Handheld technologies

use a panel that carries an electrical charge. When a finger touches the screen, the

touch disrupts the panel's electrical field. The disruption is registered and sent to the

software, which then initiates a response to the gesture.

6. IMAGE CAPTURING [10]

Image Capture is an application program that enables users to upload pictures from

digital cameras or scanners which are either connected directly to the computer or the

network. It provides no organizational tools like Photo but is useful for collecting

pictures from a variety of sources with no need for drivers. It achieves this by

receiving the picture as it is and through a conversion process, downloads it onto your

computer. It uses QuickTime Image Capture. This way it doesn't need to understand

anything about the actual camera to be able to do that. It was first introduced in Mac

OS X version 10.0(cheetah).



Fig.11 The user taking making a pose gesture [19]

Fig.12 The camera takes the photo of the location [19]

6.1 IMAGE CAPTURE API

Image Capture is also the name of the Application Programming Interface associated

with the Image Capture Application. Drivers or software built to use this interface can

also automatically interface many imaging systems and software that run on Mac OS

X. [19]

6.2 FIDUCIAL

A fiduciary marker or fiducial is an object used in the field of view of an imaging

system which appears in the image produced, for use as a point of reference or a

measure. It may be either something placed into or on the imaging subject, or a mark



or set of marks in the reticle of an optical instrument. In physics, 3D computer

graphics and photography fiducials are reference points: fixed points or lines within a

scene to which other objects can be related or against which objects can be measured.

Fiduciary markers are used in a wide range of medical imaging applications. Images

of the same subject produced with two different imaging systems may be correlated

by placing a fiduciary marker in the area imaged by both systems. In this case, a

marker which is visible in the images produced by both imaging modalities must be

used. [ 7]

7.0TECHNOLOGIE’S THAT ARE RELATED TO SIXTH SENSE

DEVICES

7.1 AUGMENTED REALITY [13]

The augmented Reality is a visualization technology that allows the user to experience

the virtual experience added over real world in real time.With the help of advanced

AR technology the information about the surrounding real world of the user becomes

interactive and digitally usable. Artificial information about the environment and the

objects in it can be stored and retrieved as an information layer on top of the real

world view. When we compare the spectrum between virtual reality, which creates

immersive, computer-generated environments, and the real world, augmented reality

is closer to the real world. Augmented reality adds graphics, sounds, hap tic feedback

and smell to the natural world as it exists. Both video games and cell phones are

driving the development of augmented reality. The augmented systems will also

superimpose graphics for every perspective available and try adjust to every

movement of the user's head and eyes. The three basic components of an augmented

reality system are the head-mounted display, tracking system and mobile computer for

the hardware. The main goal of this new technology is to merge these three

components into a highly portable unit much like a combination of a high tech

Walkman and an ordinary pair or eyeglasses. The head-mounted display used in

augmented reality systems will enable the user to view superimposed graphics and

text created by the system. Another component of an augmented reality system is its



tracking and orientation system. This system pinpoints the user's location in reference

to his surroundings and additionally tracks the user's eye and head movements.

Augmented reality systems will need highly mobile computers. As of now many

computers aren’t there to satisfy to provide this option.

7.2GESTURE RECOGNIZATION

It is a technology which is aimed at interpreting human gestures with the help of

mathematical algorithms. Gesture recognition technique basically focuses on the

emotion recognition from the face and hand gesture recognition. Gender recognition

technique enables humans to interact with computers in a more direct way without

using any external interfacing devices. It can provide a much better alternative to text

user interfaces and graphical user interface which requires the need of a keyboard or

mouse to interact with the computer. Interfaces which solely depends on the gestures

requires precise hand pose tracking. In the early versions of gesture recognition

process special type of hand gloves which provide information about hand position

orientation and flux of the fingers. In the Sixth Sense devices coloured bands are used

for this purpose. Once hand pose has been captured the gestures can be recognised

using different technique’s. Neural network approaches or statistical templates are the

commonly used techniques used for the recognition purposes. This technique have an

high accuracy usually showing accuracy of more than 95%. Time dependent neural

network will also be used for real time recognition of the gestures.

7.3 COMPUTER VISION [4]

Computer vision is the technology in which machines are able to interpret/extract

necessary information from an image. Computer vision technology includes various

fields like image processing, image analysis and machine vision. It includes certain

aspect of artificial intelligence techniques like pattern recognition. The machines

which implement computer vision techniques require image sensors which detect

electromagnetic radiation which are usually in the form of ultraviolet rays or light

rays. The computer vision find itself applicable in varies field of interest. One such



field is bio medical image processing. It’s also used in autonomous vehicle like

SUV’s. The computer vision technique basically includes four processes.

1. Recognition: One of the main task of computer vision technique is to determine

whether the particular object contain the useful data or not.

2. Motion Analysis: Motion analysis includes several tasks related to estimation of

motion where an image sequence is processed continuously to detect the velocity at

each point of the image or in the 3D scene.

3. Scene Reconstruction: Computer vision technique employs several methods to

recreate a 3D image from the available images of a scene.

4. Image Restoration: The main of aim of this step is to remove noise from an given

image. The simplest method involves using simple filters like low pass or median

filters. In order to get better quality images more complex methods like In painting

are used.

7.3.1 APPLICATION FOR COMPUTER VISION

One of the most prominent application fields is medical computer vision or

medical image processing. This area is characterized by the extraction of

information from image data for the purpose of making a medical diagnosis of

a patient. Generally, image data is in the form of microscopy images, X-ray

images, angiography images, ultrasonic images, and tomography images. An

example of information which can be extracted from such image data is

detection of tumors, arteriosclerosis or other malign changes.

A second application area in computer vision is in industry, sometimes called

machine vision, where information is extracted for the purpose of supporting a

manufacturing process. One example is quality control where details or final

products are being automatically inspected in order to find defects. Another

example is measurement of position and orientation of details to be picked up

by a robot arm.

Military applications are probably one of the largest areas for computer vision.

The obvious examples are detection of enemy soldiers or vehicles and missile


http://en.wikipedia.org/wiki/Arteriosclerosis


guidance. More advanced systems for missile guidance send the missile to an

area rather than a specific target, and target selection is made when the missile

reaches the area based on locally acquired image data. In this case, automatic

processing of the data is used to reduce complexity and to fuse information

from multiple sensors to increase reliability.

One of the newer application areas is autonomous vehicles, which include

submersibles, land-based vehicles (small robots with wheels, cars or trucks),

aerial vehicles, and unmanned aerial vehicles (UAV).The level of autonomy

ranges from fully unmanned vehicles to vehicles where computer vision based

systems support a driver or a pilot in various situations. [4]

8. PRO’S OF SIXTH SENSE TECHNOLOGY [14]

Easy to Draw: A person can draw anything on paint with the help of his

fingers. So, why to use mouse for painting.

Capture photos with fingers: why to take camera on your holiday and no

tension for the photo space as this Sixth Sense computer will work like

your camera. It captures the photo, when you make a square with your

fingers, highlighting which one you want to frame.

Phone Call: You can call to your friend by typing the numbers on your

hand. It displays the keypad of the phone over your palm and the keys

appear on the four finger. Use your another hand's finger to press the

keys.

Check the brand of the Product: It helps you to choose the best brand

product from the super market.

Read Books easily: Check out the ratings of the Book you are going to

buy, it checks the ratings from the internet. And another amazing thing is

that it reads the book for you.

Newspapers: Did you saw the moving pictures of the Newspaper in the

movie Harry Potter; it is quite similar to it. It Searches the most

appropriate video from the web by seeing the headlines or the caption of

the News report.



Check your Flight Status: You can check the status of the flight while

you are on Taxi. Just place the ticket in front of the projector and it checks

its status from the internet.

9. APPLICATIONS OF SIXTH SENSE TECHNOLOGY [19]

The Sixth Sense prototype implements several applications that demonstrate the

usefulness, viability and flexibility of the system.

The map application lets the user navigate a map displayed on a nearby

surface using hand gestures, similar to gestures supported by multi-touch

based systems, letting the user zoom in, zoom out or pan using intuitive hand

movements

Fig.13: Map Application [5]

Fig.14: Drawing Application [5]



• The drawing application lets the user draw on any surface by tracking the fingertip

movements of the user’s index finger. Sixth Sense also recognizes user’s freehand

gestures (postures). For example, it implements a gestural camera that takes photos of

the scene the user is looking at by detecting the ‘framing’ gesture. The user can stop

by any surface or wall and flick through the photos he/she has taken. [8]

Sixth Sense also lets the user draw icons or symbols in the air using the

movement of the index finger and recognizes those symbols as interaction

instructions. For example, drawing a magnifying glass symbol takes the user

to the map application or drawing a ‘@’ symbol lets the user check his mail.

The Sixth Sense system also augments physical objects the user is interacting

with by projecting more information about these objects projected on them.

For example, a newspaper can show live video news or dynamic information

can be provided on a regular piece of paper. The gesture of drawing a circle on

the user’s wrist projects an analog watch.

Fig.15: Drawing a circle on the user’s wrist projects an analog watch. [5]

Make a call by Extend hand on front of the projector and numbers will appear

for to click. Know the time by Draw a circle on wrist and a watch will appear.

take a photo by Just make a square with fingers, highlighting what want to

frame, and the system will make the photo—which can later organize with the

others using own hands over the air.



And the device has a huge number of applications, it is portable and easily to

carry as can wear it in neck. The drawing application lets user draw on any

surface by observing the movement of index finger. Mapping can also be done

anywhere with the features of zooming in or zooming out. The camera also

helps user to take pictures of the scene is viewing and later can arrange them

on any surface.[8]

Some of the more practical uses are reading a newspaper. Reading a

newspaper and viewing videos instead of the photos in the paper. Or live

sports updates while reading the newspaper.

Fig.16: News paper [5]

The device can also tell arrival, departure or delay time of air plane on tickets.

For book lovers it is nothing less than a blessing. Open any book and find the

Amazon ratings of the book. To add to it, pick any page and the device gives

additional information on the text, comments and lot more add on feature.

However, instead of requiring you to be in front of a big screen like Tom

Cruise, Sixth Sense can do its magic and a lot more everywhere, even while

you are jumping hysteric over Oprah's sofa. The camera recognizes objects

around you instantly, with the micro-projector overlaying the information on

any surface, including the object itself or your hand. Then, you can access or

manipulate the information using your fingers.

Need to know the time? Draw a circle on your wrist and a watch will appear.

Want to take a photo? Just make a square with your fingers, highlighting what

you want to frame and the system will make the photo which you can later



organize with the others using your own hands over the air. But those are just

novelty applications.

Need to make a call? Extend your hand on front of the projector and numbers

will appear for you to

Fig.17: Numbers on Hand to make a Call [17]

The true power of Sixth Sense lies on its potential to connect the real world

with the Internet, and overlaying the information on the world itself. Imagine

you are at the supermarket, thinking about what brand of soap is better. Or

maybe what food you should get for tonight's dinner. Just look at objects, hold

them on your hands, and Sixth Sense will show you if it's good or bad, or if it

fits your preferences or not.

Now take this to every aspect of your everyday life. You can be in a taxi going

to the airport, and just by taking out your boarding pass, Sixth Sense will grab

real time information about your flight and display it over the ticket. You

won't need to do any action. Just hold it in front of your and it will work. [11]

The current prototype system costs approximate $350 to build. [12],[13]

The key here is that Sixth Sense recognizes the objects around you, displaying

information automatically and letting you access it in any way you want, in the

simplest way possible.

Clearly, this has the potential of becoming the ultimate "transparent" user interface for

accessing information about everything around us. If they can get rid of the colored

finger caps and it ever goes beyond the initial development phase, that is. But as it is



now, it may change the way we interact with the real world and truly give everyone

complete awareness of the environment around us. [11]

10. WHY SIXTH SENSE TECHNOLOGY?

Humans take decisions after acquiring inputs from the senses. But the information we

collect aren’t enough to result in the right decisions. But the information which could

help making a good decision is largely available on internet. Although the information

can be gathered by connecting devices like computers and mobiles but they are

restricted to the screen and there is no direct interaction between the tangible physical

world and intangible digital world. This sixth sense technology provides us with the

freedom of interacting with the digital world with hand gestures. This technology has

a wide application in the field of artificial intelligence. This methodology can aid in

synthesis of bots that will be able to interact with humans.

11. FUTURE ENHANCEMENT

Further progress has been made with the system by broadcasting their latest

demonstration online. The new Sixth Sense system uses a camera that is hung from

the user’s neck, instead of mounted on your hat. Also hanging from the neck is a pico

projector and a mirror that will display onto any surface that the user is looking at.

When we first heard about this system, the user had to have a mini computer strapped

to their back that would always be connected to the internet and could find the

required information. The new system eliminates the backpack and instead connects

to your cell phone to retrieve information. The Sixth Sense is still in its development

stages.



Mouse less:

Fig.18 Mouse less [17]

Mouse less is a concept which is developed in order to avoid the usage of mouse in

computers. It helps to provide a familiarity of the physical mouse without actually

requiring a physical mouse.

Even though the computer hardware technology has been developed to a greater

extent, still there has been no proper substitute for a physical mouse. This new

invention aims to remove the requirement of having a physical mouse altogether but

still it provides the feeling of a physical mouse that users are familiar with. It

basically consists of an Infrared laser beam and an Infrared camera. Both laser and

camera are embedded together with the computer. The laser beam module is modified

with a line cap and placed such that it creates a plane of IR laser just above the surface

the computer sits on. The user cups their hand, as if a physical mouse was present

underneath, and the laser beam lights up the hand which is in contact with the surface.

The IR camera detects those bright IR blobs using computer vision. The change in the

position and arrangements of these blobs are interpreted as mouse cursor movement

and mouse clicks.



Third Eye:

Third Eye is a new technique that enables multiple viewers to see different things on

a same display screen at the same time.

Fig.19: Third Eye [17]

With third Eye, We can have a public sign board where a Japanese tourist sees all the

instructions in Japanese and an American in English. - We don't need to have the split

screen in games now. Each player can see his/her personal view of the game on the

TV screen. - Two people watching TV can watch their favorite channel on a single

TV screen. - A public display can show secret messages or patterns. - In the same

movie theater, people can see different end of a suspense movie.

Inktuitive:

Even though there has been a tremendous development of computer aided designing

tools, still paper and pencil is the preferred by most of the designers ,especially in the

earlier stages of designing .



Fig.19: Inktuitive [17]

The project ‘inktuitive’ is aimed to combine the process of creation that is inherent in

paper and pencil with the power of computing that the digital design tools provide. It

also extends the natural work-practice of using physical paper by giving the pen the

ability to control the design in physical three-dimensional space, freeing it from its tie

to the paper. There always remains an intuition of pen and paper ,but the lines that

have been drawn are be captured and translated into shapes in the digital world. The

physical paper is augmented with overlaid digital strokes. The platform provides a

novel interaction mechanism for drawing and designing using above the surface pen

movements. In other words ‘inktuitive’ can be considered as an intuitive physical

design workspace that aims to bridge the gap and bring together the conventional

design tools such as paper and pencil with the power and convenience of the digital

tools for design.

TaPuma:

TaPuma is the short form of Tangible Public Map. It is another attempt to bring the

digital world nearer to the real world by exploring the techniques of sixth sense

technology. Key word based search mechanism has vastly influenced and modified

the ways in which we search information’s. The keyword based search allows the user

to filter out the required data from the vast data that are available on search engines

such as yahoo and google. TaPuMa allows people to use their own belongings, the

objects they usually carry with them to access relevant and just-in-time information

and locations of places from a public map. The project also explores and analyzes the

advantages and challenges of this novel interaction mechanism, where real life objects



serve as the interface for information acquirement. The broad concept behind the

project TaPuMa is Object Amelioration, where the functions of everyday objects can

be expanded by using their affordances or functionalities in a variety of different

contexts.

CONCLUSION



Technology has transformed the world in which we live, changing how we spend our

time, how we understand ourselves, and how we interact with others. Technological

innovation results in social and economic change. Thus, Sixth Sense Technology will

lead to the development of present world. And it is the present world that promises to

restructure human life and activity.

REFERENCES



1. http://pdfdatabase.com/sixth-sense-technology-in-technical.html

2. http://microreaders.blogspot.com/2009/11/sixth-sense-technology-shows-that-

we.html

3. http://ghallabhansali.com

4. http://fluid.media.mit.edu/assets/_pubs/wip177-mistry.pdf

5. http://techinfo.org/5983/Gesture_recognition

6. http://www.pranavmistry.com/projects/sixthsense

7. http://techhost.edu/tech/Fiduciary_marker

8. http://www.college-seminars.com/sixth-sense-technology

9. http://gizmodo.com/5167790/sixth-sense-technology-may-change-how-we-

look-at-the-world-forever

10. http://www.pranavmistry.com/projects/sixthsense/

11. http://er-divya-pai.blogspot.in/2013/01/pranav-mistry-innovator-of-sixth-

sense.html

12. http://www.ted.com/talks/

pranav_mistry_the_thrilling_potential_of_sixthsense_technology.htm

13. http://www.commoncraft.com/augmented-reality

14. http://www.arcane-technologies.com/en/

15. http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html

16. http://timesofindia.indiatimes.com/Health__Science/

Sixth_sense_gizmo_a_sci-fi_reality/articleshow/4083978.cms

17. http://www.seminarpapers.net/sixth-sense.pdf


http://timesofindia.indiatimes.com/Health__Science/Sixth_sense_gizmo_a_sci-fi_reality/articleshow/4083978.cms

http://timesofindia.indiatimes.com/Health__Science/Sixth_sense_gizmo_a_sci-fi_reality/articleshow/4083978.cms

http://www.arcane-technologies.com/en/

http://er-divya-pai.blogspot.in/2013/01/pranav-mistry-innovator-of-sixth-sense.html

http://er-divya-pai.blogspot.in/2013/01/pranav-mistry-innovator-of-sixth-sense.html

http://www.pranavmistry.com/projects/sixthsense/

http://gizmodo.com/5167790/sixth-sense-technology-may-change-how-we-look-at-the-world-forever

http://gizmodo.com/5167790/sixth-sense-technology-may-change-how-we-look-at-the-world-forever

http://www.college-seminars.com/sixth-sense-technology

http://techhost.edu/tech/Fiduciary_marker

http://www.pranavmistry.com/projects/sixthsense

http://fluid.media.mit.edu/assets/_pubs/wip177-mistry.pdf

http://www.ghallabhansali.com/

http://microreaders.blogspot.com/2009/11/sixth-sense-technology-shows-that-we.html

http://microreaders.blogspot.com/2009/11/sixth-sense-technology-shows-that-we.html

http://pdfdatabase.com/sixth-sense-technology-in-technical.html

sixth sense technology 04

Technology

sense technology works

sense attempts

sense technology3

physical objects

projected information

projects information

digital information

physical world