GESTICULATION

Report submitted in partial fulfillment

Of the requirement for the degree of

B.Tech.

In

Information Technology Engineering

Under the Supervision of

Ms. Parul Yadav

By

Yogesh Madaan (0411503112)

Kanika Agarwal (05811503112)

To

Bharati Vidyapeeth’s College of Engineering

Paschim Vihar East, New Delhi

Of

Guru Gobind Singh Indraprastha University

Dwarka, New Delhi

Year 2015-2016

1

DECLARATION

This is to certify that Report entitled Gesticulation, which is submitted by us in

partial fulfillment of the requirement for the award of degree B.Tech. in Bharati

Vidyapeeth’s College of Engineering, Paschim Vihar East, New Delhi, of GGSIP

University, Dwarka, New Delhi, comprises only our original work and due

acknowledgement has been made in the text to all other material used.

DATE: NAME OF STUDENTS:

Yogesh Madaan (0411503112)

Kanika Agarwal (05811503112)

APPROVED BY Prof. (Dr.) Vanita Jain

(Head of Department, Information Technology

Bharati Vidyapeeth’s College of Engineering)

CERTIFICATE

2

This is to certify that Report entitled Gesticulation, which is submitted, by Yogesh

Madaan and Kanika Agarwal in partial fulfillment of the requirement for the award

of degree B.Tech. in Information Technology Engineering to BVCOE, Paschim Vihar

East, New Delhi, of GGSIP University, Dwarka, New Delhi is a record of the

candidates’ own work carried out by them under my supervision. The matter

embodied in this report is original and has not been submitted for the award of any

other degree.

DATE: SUPERVISOR:

Ms. Parul Yadav

3

ABSTRACT

The Internet of Things (IoT) is – at its most profound levels – about creating digital

representations of real-world objects. It is a phenomenon that draws on rapid

developments within IT to spark insights and to help companies create entirely new

types of services and business areas. Today Devices are becoming more capable and

more broadly utilized, and when they are connected to the internet, they are integrated

into vast numbers of different applications across sectors. Hence, IoT is evolving. [1]

In this project, an effort has been made to implement the above technology that is IoT,

using a hardware device called Myo Gesture Control Armband. This hardware device,

developed by Thalmic Labs, senses the motion of an arm of a human being and acts

accordingly. This project has three modules- The Windows System Module, The

Android OS Module and The Home Automation. These modules involve the

integration of the hardware device with an existing application on a windows system,

a special application development on an android OS supported mobile device, and the

operation of basic home appliances with the hardware device respectively.

4

TABLE OF CONTENTS

Title Page………………………………………………………………........................i

Declaration.....................................................................................................................ii

Certificate………………………… ……………………………………………..…...iii

Abstract……………………………………………………………………..………...iv

Table of Contents…………………………………………………..………………….v

List of Figures……………………………………………………..…………………vii

1. Introduction………………………....……………………………………..………81.1. Overview……………………………………………………………………...81.2. Existing System……. …………………………………………......................81.3. Scope of Proposed Project…………………………………………………....9

2. Pre-Requisite Technologies………………………………………………………102.1. Internet of Things……………………………………………………………102.2. Myo Gesture Control Armband……………………………………………..122.3. Bluetooth Low Energy………………………………………………………12

2.3.1. BLE Platform Support ……………………………………………….132.3.2. Technical Details……………………………………………………..14

2.4. Arduino……………………………………………………………………...152.4.1. Technical Details…………………………………………………….16

3. Myo Armband………………………………………………………………....…173.1. Design……………………………………………………………………….183.2. Setup and Calibration……………………………………………………….193.3. Performance and use………………………………………………………..203.4. Compatibility………………………………………………………………..213.5. Features……………………………………………………………………...213.6. Highs and Lows……………………………………………………………..22

4. Gesticulation…………………………………………………………………..…234.1. Idea………………………………………………………………………….234.2. Project Phases…………………………………………………………….…23

4.2.1. Phase 1……………………………………………………………….234.2.2. Phase 2……………………………………………………………….244.2.3. Phase 3……………………………………………………………….24

5. Selfie Camera and Gallery Viewer.……………………………………………..256. Smart Assistant……………………………………………………………….…277. Home Automation (Prototype)………………………………………………….28

5

8. Summary………………………………………………………………………..309. Conclusion……………………………………………………………………....3110. Future Scope…………………………………………………………………….3211. References………………………………………………………………………3312. Appendix………………………………………………………………………..34

6

LIST OF FIGURES

Fig.1 Type of Gestures of Myo…………………………..............…………………17

Fig.2 Hardware parts of Myo……………………………………………………….19

Fig.3 Flow diagram for windows’ camera application……………………………...26

Fig.4 Flow diagram for android utility application…………………………………27

Fig.5 Flow diagram for Home Automation………………………………………...29

INTRODUCTION

7

The objective of our project is to use technology for a social change. We have

tried to design a platform and set of utility tools that can be used by people

with visual impairment to help them in day to day life with a help of just

smartphones and reduce their effort. So with the vocabulary meaning of

gesticulation task performance can be accomplished with just a set of gestures.

With the initial idea of using a device to create interfaces for youngsters to

control their smart devices from a distance without physical contact with the

device, we are now building an aid to serve as a partner for blind and

physically handicapped persons.

1.1 Overview

Today, there are the inventors who work on various hardware devices with

some other hardware devices to make a new one. There are coders,

programmers, software developers, designers who are bringing new innovative

ideas and revolutionizing the world of web. There are brilliant hardware

devices, amazing software; the need of hour is to bring together the two and

kick-start a new era of IoT that is Internet of Things.

An effort has to be made to implement the above technology that is IoT. A hardware

device called MYO, developed by Thalmic Labs, can be used for the same. This

device senses the motion of an arm of a human being and acts accordingly. This

feature of MYO can be used in integration with various software platforms to develop

an innovation. These platforms may include windows, android OS and even other

hardware technology like Arduino.

1.2 Existing System

For the integration of MYO with the windows system, The Thalmic Labs has

provided the whole MYO market where the user can use any utility application

connector he/she wants to operate with the gesture of a hand. For example,

PowerPoint connector is used to switch between the slides of a presentation

with their hand gesture. The user has to download the connector from the MYO

market and add it in the application manager of for MYO, that is the MYO

Connect. The MYO connect is the windows application that acts as an interface

8

between the connector and the other windows application. The Thalmic Labs

also provide the facility for the developers. People can code their own

application connector and use for their application using the MYO Connect.

Just like for windows, The Thalmic Labs also provides application for MYO in

android for the mobile devices. Some of these applications are MYO music and

MYO dialer. It also provides developer options for android. The Myo gesture

control armband is never integrated with Arduino.

1.3 Scope of Proposed Project

In the integration of Myo Gesture control Armband with the windows system a

pre-existing application, camera is used. The Myo is integrated with the camera

so that the user do not have to touch the system to click his photograph, make a

video or view the gallery. All will be done by gesturing the arm in which Myo

is worn.

For android, a special general utility mobile application is developed in which

the user can use his selfie stick to click his photograph without setting any

timer. Also, it has a feature by which the user can feel all the notification of his

phone on his arm and can act accordingly.

An effort is made to generate a basic prototype for such integration through an

android mobile application. This has been done to solve the purpose of home

automation. Through the gesture of a hand having Myo, The electric appliances

like, a tube light, a fan, etc. of a room can be controlled.

PRE-REQUISITE TECHNOLOGIES

9

2.1 Internet of Things – IoT

The Internet of Things (IoT) is the network of physical objects or "things" embedded

with electronics, software, sensors, and network connectivity, which enables these

objects to collect and exchange data. The Internet of Things allows objects to be

sensed and controlled remotely across existing network infrastructure, creating

opportunities for more direct integration between the physical world and computer-

based systems, and resulting in improved efficiency, accuracy and economic benefit.

Each thing is uniquely identifiable through its embedded computing system but is able

to interoperate within the existing Internet infrastructure.

IoT offers advanced connectivity of devices, systems, and services that goes beyond

machine-to-machine communications (M2M) and covers a variety of protocols,

domains, and applications. The interconnection of these embedded devices (including

smart objects), is expected to usher in automation in nearly all fields, while also

enabling advanced applications like a Smart Grid, and expanding to the areas such as

smart cities.

"Things," in the IoT sense, can refer to a wide variety of devices such as heart

monitoring implants, biochip transponders on farm animals, electric clams in coastal

waters, automobiles with built-in sensors, or field operation devices that assist

firefighters in search and rescue operations. These devices collect useful data with the

help of various existing technologies and then autonomously flow the data between

other devices. Current market examples include smart thermostat systems and

washer/dryers that use Wi-Fi for remote monitoring.

Besides the plethora of new application areas for Internet connected automation to

expand into, IoT is also expected to generate large amounts of data from diverse

locations that is aggregated very quickly, thereby increasing the need to better index,

store and process such data.

Survey conducted by Mckinsey & Company in June 2015 To get a broader view of

the IoT’s potential benefits estimates that the IoT has a total potential economic

impact of $3.9 trillion to $11.1 trillion a year by 2025. At the top end, that level of

10

value—including the consumer surplus—would be equivalent to about 11 percent of

the world economy (exhibit).[1]

Achieving this kind of impact would require certain conditions to be in place, notably

overcoming the technical, organizational, and regulatory hurdles. In particular,

companies that use IoT technology will play a critical role in developing the right

systems and processes to maximize its value. Among our findings:

Interoperability between IoT systems is critical. Of the total potential

economic value the IoT enables, interoperability is required for 40 percent on

average and for nearly 60 percent in some settings.

Currently, most IoT data are not used. For example, on an oilrig that has

30,000 sensors, only 1 percent of the data are examined. That is because this

information is used mostly to detect and control anomalies—not for

optimization and prediction, which provide the greatest value.

Business-to-business applications will probably capture more value—nearly

70 percent of it—than consumer uses, although consumer applications, such as

fitness monitors and self-driving cars, attract the most attention and can create

significant value, too.

The IoT has a large potential in developing economies. Still, we estimate that

it will have a higher overall value impact in advanced economies because of

the higher value per use. However, developing economies could generate

nearly 40 percent of the IoT’s value, and nearly half in some settings.

Customers will capture most of the benefits. We estimate that IoT users

(businesses, other organizations, and consumers) could capture 90 percent of

the value that IoT applications generate. For example, in 2025 remote

monitoring could create as much as $1.1 trillion a year in value by improving

the health of chronic-disease patients.

11

A dynamic industry is evolving around IoT technology. As in other

technology waves, both incumbents and new players have opportunities.

Digitization blurs the lines between technology companies and other types of

businesses; makers of industrial machinery, for example, are creating new

business models by using IoT links and data to offer their products as a

service.[1]

2.2 Myo Gesture Control Armband

Myo Gesture Control Armband, launched in June 2013, is a wearable technology

developed by a Canada based start-up named Thalmic Labs. Using a technique known

as electromyography, this little armband is able to read electrical signals from the

muscles in the forearm, map them to gestures made with the hand, and control other

devices with those gestures. The Myo armband gives a touch-free control of

technology with hand gestures and motion. It aims to give a motion control without

the need for a Wii remote, a presentation controller, or anything else that has to be

held in hand. Instead, the armband just has to be slipped like a blood pressure sleeve

(or a Nintendo Power Glove) and things can be controlled with the movements of

fingers and wrist. Most gesture-control systems, like Microsoft’s Kinect, still detect

movements with cameras, which can be thrown off by poor lighting conditions,

distance, and simple obstructions. By drawing gesture information directly from the

arm muscles instead of a camera, Myo circumvents all these problems — and works

with devices that do not have a camera in the first place. It detects five distinct hand

gestures to wirelessly control tech, like music, games, and presentation slides, all with

the flick of a wrist.

Myo wraps it in eight different blocks, each of which contains a medical-grade EMG

sensor. The armband also uses a three-axis gyroscope, three-axis accelerometer, and

three-axis magnetometer to sense motion in any direction.

2.3 Bluetooth Low Energy

Bluetooth Low Energy (BLE), sometimes referred to as "Bluetooth Smart", is a light-

weight subset of classic Bluetooth and was introduced as part of the Bluetooth 4.0

12

core specification. While there is some overlap with classic Bluetooth, BLE actually

has a completely different lineage and was started by Nokia as an in-house project

called 'Wibree' before being adopted by the Bluetooth SIG.

There are plenty of wireless protocols out there for engineers and product designers,

but what makes BLE so interesting is that it's almost certainly the easiest way to

design something that can talk to any modern mobile platform out there (iOS,

Android, Windows phones, etc.), and particularly in the case of Apple devices it's the

only HW design option that doesn't require you to jump through endless hoops to be

able to legally market your product for iOS devices.

This guide will give you a quick overview of BLE, specifically how data is organized

in Bluetooth Low Energy, and how devices advertise their presence so that you can

connect to them and start passing data back and forth.

2.3.1 BLE Platform Support

Support for Bluetooth 4.0 and Bluetooth Low Energy (which is a subset of BT 4.0) is

available on most major platforms as of the versions listed below:

iOS5+ (iOS7+ preferred)

Android 4.3+ (numerous bug fixes in 4.4+)

Apple OS X 10.6+

Windows 8 (XP, Vista and 7 only support Bluetooth 2.1)

GNU/Linux Vanilla BlueZ 4.93+

Bluetooth Smart extends the use of Bluetooth wireless technology to devices that are

powered by small, coin-cell batteries such as watches and toys. Other devices such as

sports & fitness, health care, keyboards and mice, beacons, wearables and

entertainment devices are enhanced by this version of the technology. In many cases,

it makes it possible to operate these devices for more than a year without recharging.

As with previous versions of the specification, the range of the radio may be

optimized according to application. The majority of Bluetooth devices on the market

today include the basic 30 foot, or 10 meter, range of the Classic Bluetooth radio, but

there is no limit imposed by the Specification. With Bluetooth Smart, manufacturers

13

may choose to optimize range to 200 feet and beyond, particularly for in-home sensor

applications where longer range is a necessity.

Bluetooth Smart features provides:

Ultra-low peak, average and idle mode power consumption

Ability to run for years on standard coin-cell batteries

Lower implementation costs

Multi-vendor interoperability

Enhanced range

This enhancement to the Bluetooth Core Specification allows two types of

implementation, dual-mode and single-mode. In a dual-mode implementation,

Bluetooth low energy functionality is integrated into an existing Classic Bluetooth

controller. The resulting architecture shares much of Classic Bluetooth technology's

existing radio and functionality resulting in a minimal cost increase compared to

Classic Bluetooth technology. Additionally, manufacturers can use current Classic

Bluetooth technology (Bluetooth v2.1 + EDR or Bluetooth v3.0 + HS) chips with the

new low energy stack, enhancing the development of Classic Bluetooth enabled

devices with new capabilities.

Single-mode chips, which will enable highly integrated and compact devices, will

feature a lightweight Link Layer providing ultra-low power idle mode operation,

simple device discovery, and reliable point-to-multipoint data transfer with advanced

power-save and secure encrypted connections at the lowest possible cost. The Link

Layer in these controllers will enable Internet connected sensors to schedule

Bluetooth low energy traffic between Bluetooth transmissions.

2.3.2 Technical Details

Data Transfers – Bluetooth Smart (low energy) supports very short data

packets (8 octet minimum up to 27 octets maximum) that are transferred at 1

Mbps. All connections use advanced sniff-sub rating to achieve ultra low duty

cycles

Frequency Hopping – Bluetooth Smart (low energy) uses the adaptive

frequency hopping common to all versions of Bluetooth technology to

14

minimize interference from other technologies in the 2.4 GHz ISM Band.

Efficient multi-path benefits increase the link budgets and range

Host Control – Bluetooth Smart (low energy) places a significant amount of

intelligence in the controller, which allows the host to sleep for longer periods

of time and be woken up by the controller only when the host needs to

perform some action. This allows for the greatest current savings since the

host is assumed to consume more power than the controller

Latency - Bluetooth Smart (low energy) can support connection setup and data

transfer as low as 3ms, allowing an application to form a connection and then

transfer authenticated data in few milliseconds for a short communication

burst before quickly tearing down the connection

Range – Increased modulation index provides a possible range for Bluetooth

Smart (low energy) of over 100 meters

Robustness – Bluetooth Smart (low energy) uses a strong 24 bit CRC on all

packets ensuring the maximum robustness against interference

Strong Security – Full AES-128 encryption using CCM to provide strong

encryption and authentication of data packets

Topology – Bluetooth Smart (low energy) uses a 32 bit access address on

every packet for each slave, allowing billions of devices to be connected. The

technology is optimized for one-to-one connections while allowing one-to-

many connections using a star topology. [4]

2.4 Arduino

The Micro is a microcontroller board based on the ATmega32U4, developed in

conjunction with Adafruit. It has 20 digital input/output pins (of which 7 can be used

as PWM outputs and 12 as analog inputs), a 16 MHz crystal oscillator, a micro USB

connection, an ICSP header, and a reset button. It contains everything needed to

support the microcontroller; simply connect it to a computer with a micro USB cable

to get started. It has a form factor that enables it to be easily placed on a breadboard.

The Micro board is similar to the Arduino Leonardo in that the ATmega32U4 has

built-in USB communication, eliminating the need for a secondary processor. This

allows the Micro to appear to a connected computer as a mouse and keyboard, in

15

addition to a virtual (CDC) serial / COM port. It also has other implications for the

behavior of the board; these are detailed on the getting started page.

2.4.1 Technical specs

Microcontroller ATmega32U4

Operating Voltage 5V

Input Voltage (recommended) 7-12V

Input Voltage (limit) 6-20V

Digital I/O Pins 20

PWM Channels 7

Analog Input Channels 12

DC Current per I/O Pin 20 mA

DC Current for 3.3V Pin 50 mA

Flash Memory 32 KB (ATmega32U4)of which 4 KB used by bootloader

SRAM 2.5 KB (ATmega32U4)

EEPROM 1 KB (ATmega32U4)

Clock Speed 16 MHz

Length 48 mm

Width 18 mm

Weight 13 g

16

MYO ARMBAND

Myo Gesture Control Armband, launched in June 2013, is a wearable technology

developed by a Canada based start-up named Thalmic Labs. Using a technique known

as electromyography, this little armband is able to read electrical signals from the

muscles in the forearm, map them to gestures made with the hand, and control other

devices with those gestures. The Myo armband gives a touch-free control of

technology with hand gestures and motion. It aims to give a motion control without

the need for a Wii remote, a presentation controller, or anything else that has to be

held in hand. Instead, the armband just has to be slipped like a blood pressure sleeve

(or a Nintendo Power Glove) and things can be controlled with the movements of

fingers and wrist. Most gesture-control systems, like Microsoft’s Kinect, still detect

movements with cameras, which can be thrown off by poor lighting conditions,

distance, and simple obstructions. By drawing gesture information directly from the

arm muscles instead of a camera, Myo circumvents all these problems — and works

with devices that do not have a camera in the first place. It detects five distinct hand

gestures to wirelessly control tech, like music, games, and presentation slides, all with

the flick of a wrist.

Fig. 1

17

Myo wraps it in eight different blocks, each of which contains a medical-grade EMG

sensor. The armband also uses a three-axis gyroscope, three-axis accelerometer, and

three-axis magnetometer to sense motion in any direction.

Muscle activity and motion readings are handled by an onboard ARM Cortex M4

Processor, which communicates with your devices via Bluetooth. For devices that do

not have Bluetooth functionality built in, Myo also comes with a Bluetooth dongle

that plugs into any USB port.

Details on battery size are not readily available, but Thalmic claims the lithium-ion

cell inside is big enough to keep Myo going for a full day of continued use.

One can also map gestures to key strokes for customized control, or dig into the open

API and free, easy-to-use SDK to create scripts and applications that put technology

at the fingertips. The device already has a sizable library of apps and plugins to offer.

So far, the Armband has been used to serve the purpose PowerPoint

presentations and media playback for apps like Netflix, Windows Media Player,

and iTunes. [2]

3.1Design

Myo has undergone some major design changes since it was first conceived, but the

final version is about as polished as you can hope for. It is relatively slim and sleek

considering how many sensors are stuffed inside its hull, and despite being a

somewhat geeky piece of gear; it does have an undeniably badass look to it.

The band itself is a series of plastic, rectangular "pods," held together by a

perimeter of rubber lining. On the underside of each pod there are three

medical grade stainless steel EMG sensors that detect the electric impulses in

the arm's muscles. The design allows the Armband to fit a multitude of

differently sized forearms.

18

Fig. 2

One of the pods sports two indicator LEDs. One is simply for the Myo logo,

while the other, rectangular light turns green to indicate the band is charging,

and light blue when it is on and connected.  

As for the cold, hard measurements, the Myo comes in at 7.5 inches for

circumference, expandable to 13 inches. It weighs only 3.3 ounces, and each

individual pod has a thickness of 0.45 inches. It is available in black or white.

[3]

3.2 Setup and calibration

Setting up Myo is a strange and unfamiliar process, but despite being unlike you’ve

ever used before, getting the armband up and running is pretty straightforward.

Once you’ve got it all charged up (via the included micro USB cable), all that’s left to

do is establish a Bluetooth connection with your computer and run through the

calibration process. Thalmic has put together a series of short videos to guide you

through the process, which make it a breeze to get going.

19

After you slip Myo onto your arm, you’ll need to wait for a couple minutes for the

sensors to warm up. The setup video explains that during this time, Myo is forming a

strong electrical connection with the muscles in your forearm. After this bond is

established, Myo vibrates to let you know it’s ready for sync.

To sync Myo and start using it, you need to perform Thalmic’s special “sync gesture,”

which involves holding your arm parallel to the ground, rotating your wrist outward,

and then swinging your forearm out 90 degrees like it’s on a hinge.

If this gesture works and Myo vibrates again, you’re off to the races.

3.3 Performance and use

Simply put, Myo’s gesture-recognition capabilities are everything they’re chalked up

to be — so long as you have the sensors calibrated properly. That’s the trick. It might

take a couple tries to get everything perfect, but once you do, Myo works like a

dream.

The first thing we tried out was Thalmic’s presentation demo, which allows you to

navigate through slideshows using gesture controls like tapping fingers together,

waving your hand right or left, and opening and closing your fist. At first we had

some trouble getting finger taps to register unless they were a bit exaggerated, but

after a quick recalibration, Myo recognized all our motions instantaneously, even if

the taps were light and quick.

After that, we tried it out on iTunes. It was awesome. Wave left and right to skip

tracks, play and pause by spreading your fingers, and roll your fist to crank up the

volume. Even when we were across the room, the gesture commands were recognized

on the first try every time, and their corresponding action was executed without any

sort of delay. So long as you’re within Bluetooth range of whatever you’re

controlling, Myo is extremely responsive.

If there’s a downside to all this, it’s probably the fact that, in its current state, Myo can

only recognize five different gestures and two types of motion. It’s not quite as

dynamic as we had initially thought, and can only read finger taps, left or right waves,

opening your hand, making a fist, and any up, down, left, right, and roll movements.

20

The sensors are likely capable of more –like, say, a “trigger pull” motion you make

with your index finger, or pointing with two fingers instead of one– but right now it’s

limited to the aforementioned gestures and motions. That’s not to say Myo feels

incomplete or incapable in any way, though. You can do a lot with the available

commands — we just expected more gestures to be available out of the box.

3.4 Compatibility

Flipping through your iTunes playlists and Keynote presentations is just the

beginning. Thalmic put Myo in the hands of developers long before the general

release, so now that it’s officially launched, the device already has a sizable library of

apps and plugins to offer.

In the Myo Marketplace, you can download integrations for all kinds of apps, games,

and utilities — Netflix, Spotify, YouTube, Minecraft, Civilization V, Kerbal Space

Program, and tons more. There’s even one that allows you to control a Parrot AR

drone like they showed in the pitch video two years ago.

Thalmic doesn’t really place any limits on how Myo can be used or what it can be

used for, so developers have basically run wild with it – which is awesome, but you’ll

definitely find that some of the integrations are much more practical than others.

One of the first applications I downloaded after setup was the Mac Desktop Utility,

which allows you to do things like switch between desktops, take a screenshot, or

show OS X’s Launchpad by using gestures. Cool, no doubt, but when you’ve already

got a keyboard and a trackpad at your fingertips, it’s pretty unnecessary to throw

gesture control into the mix.

For other things, however, Myo makes a whole lot of sense. Linked up to media

players like iTunes or Spotify, it functions as a sort of wearable hands-free controller.

The other night I wore it while I cooked dinner and didn’t have to whip out my phone

to skip tracks on Spotify – I could just wave my hand in the air over the stove and

continue sautéing my onions.

21

3.5 Features

Along with the Armband, the whole package is also provided with a standard

micro USB cable for charging and a USB Bluetooth adapter for pairing the

device with the computer. It also comes with ten sizing clips that can be used to

readjust the band if it doesn't fit the arm properly.      

The Armband uses a three-axis accelerometer, a three-axis magnetometer, and a

three-axis gyroscope to sense motion. An onboard ARM Cortex M4 processor

talks with any connected devices via Bluetooth. The band is compatible with

PCs running Windows 7, Windows 8, and Mac OS X 10.8 and above. It also

works with Bluetooth 4.0-equipped mobile devices running iOS 7.0 and higher,

as well as Android 4.3 and above.  

According to Thalmic Labs, a built-in rechargeable lithium-ion battery can

keep the Myo going for one full day on a single charge.

3.6 Highs and Lows of the Device

Highs

Easy setup

Expandable design

Broad compatibility

Super fast and responsive

Growing library of apps/integrations

Lows

Limited range of gestures

Perfect calibration takes a few tries[3]

22

GESTICULATION

4.1 Idea

Gesticulation is a project in which an effort has been made to implement IoT using a

hardware device called Myo Gesture Control Armband. This hardware device,

developed by Thalmic Labs, senses the motion of an arm of a human being and acts

accordingly. This project has three modules- The Windows System Module, The

Android OS Module and The Home Automation Module. These modules involve the

integration of the hardware device with an existing application on a windows system,

a special application development on an android OS supported mobile device, and the

operation of basic home appliances with the hardware device respectively.

With the initial idea of using a device to create interfaces for youngsters to control

their smart devices from a distance without physical contact with the device we are

now building an aid to serve as a partner for blind and physically handicapped

persons.

The idea is to use technology for a social change. We have tried to design a platform

and set of utility tools that can be used by people with visual impairment to help them

in day to day life with a help of their smartphones and this gesture control hardware

device, MYO and reduce their effort. So now various tasks of everyday life can be

accomplished with just a set of a few hand gestures.

4.2 Project Phases

4.2.1 Phase 1

The idea is very simple and unique with try to do ordinary task in a creative and smart

way using technology and gadgets. With the help of utility camera a gallery can be

smartly controlled from you hands from a distance apart removing the difficulty of

switching images/ slides during a slide show. The mobile version of the application is

targeting the youngster that are normally seen capturing a selfie using a stick which

can now only be completed by putting a phone at a distance and opening your fist.

23

4.2.2 Phase 2

The smart application is always running with you establishing a smart and battery

efficient Bluetooth connection with you smart device to assist you immediately. The

application gets triggered whenever some receiver is triggered.. The functionalities of

the application includes

• Handle various receivers of the mobile (Call Receiver, Message receiver)

• Accept and reject calls from a distance

• Set phone to speaker mode

• Read out incoming text messages loudly

• Handle notifications

4.2.3 Phase 3

Continuing with the most basic and commonly used IOT idea of making your home

smart we have designed a prototype to help unpaired people in controlling their home

appliances remotely with their virtual presence. The is just a prototype and we are

working on the research for this is order to create a stable product. The idea is to

create switch boards that have integrated wifi modules in them that will be handled by

a home server .The home server in return will be connected with various clients

(Android phone in our case) . The armband will be in continuous connection with the

app and will interact with the switch relay to control the appliances.

24

SELFIE CAMERA AND GALLERY VIEWER

The idea is very simple and unique with an effort made to be able to do

ordinary task in a creative and smart way using technology and gadgets. With

the help of this module, utility camera and gallery can be smartly controlled by

your hands from a distance apart removing the difficulty of switching images/

slides during a slide show. The mobile version of the application is targeting

the youngster that are normally seen capturing a selfie using a stick, which can

now be completed by putting a phone at a distance and opening your fist.

In the windows version of this module the effort is made to allow the user to

use the camera of the system without actually touching the system. All they

have to do is look into the camera, smile, twist the hand a little and the photo is

clicked. Through this tool the user can also make a video, can look into the

gallery of all his photos, play a video, increase and decrease the brightness of

the image before clicking and much more.

The module is providing support for two different type of platforms namely

Windows operating system and Android operating system. In windows system

default application “Camera.exe” and “Photos.exe” will be used and we have

provided MYO support by developing a connector which runs in background

under the surveillance of Myo Connect which handles the direct connection

with the MYO armband and manages its different connections states and

different pose actions. The connector gets notified about the change of states of

armband and acts accordingly. With mapping of various keyboard keys and

mouse controls the application is controlled with the hand gestures.

The working in android application is slightly different in case of mobile

application where direct connection is established between the mobile

application and MYO armband. Thalmic labs have provided the OS SDK to

25

interact directly with the armband. In order to have an efficient performance in

terms of battery BLE(Bluetooth low energy ) connection has been established

rather than the classical Bluetooth connection between the armband and the

application.

Fig. 3

26

SMART ASSISTANT

This module is based only on Android platform.

The smart application is always running with you establishing a smart and

battery efficient Bluetooth connection with your smart device to assist you

immediately. The application gets triggered whenever some receiver is

triggered. The functionalities of the application include

• Handle various receivers of the mobile (Call Receiver, Message

receiver)

• Accept and reject calls from a distance

• Set phone to speaker mode

• Read out incoming text messages loudly

• Handle notifications

Moving on the technical specification of the application it has support same as

BLE for android OS so it will be supported for Android OS version greater than

4.3 thus targeting the 76% of total android users.

Apart from call and message alerts the application will handle all the third

party notifications like WhatsApp messages and Hangout notifications. We

have designed the different broadcast receivers for different tasks. A

background service is designed which starts running automatically as soon as

phone boots up and tries to connect to the nearest armband.

27

Fig. 4

HOME AUTOMATION (PROTOTYPE)

Home automation is the use of one or more computers to control basic home functions

and features automatically and sometimes remotely. An automated home is

sometimes called a smart home .

Home automation can include the scheduling and automatic operation of water

sprinkling, heating and air conditioning, window coverings, security systems, lighting,

and food preparation appliances. Home automation may also allow vital home

functions to be controlled remotely from anywhere in the world using a computer

connected to the Internet. Besides the functions already mentioned, remote control

can be extended to telephones and answering machines, fax machines, amateur radios

and other communications equipment, and home robot s such as automatic vacuum

cleaners.

Continuing with the most basic and commonly used IOT idea of making your

home smart, we have designed a prototype to help impaired people in

controlling their home appliances remotely with their virtual presence. The is

just a prototype and we are working on the research for this in order to create a

stable product. The idea is to create switchboards that have integrated Wi-Fi

modules in them that will be handled by a home server. The home server in

return will be connected with various clients (Android phone in our case). The

28

armband will be in continuous connection with the app and will interact with

the switch relay to control the appliances.

Fig. 5

29

SUMMARY

Today, there are the inventors who work on various hardware devices with

some other hardware devices to make a new one. There are coders,

programmers, software developers, designers who are bringing new innovative

ideas and revolutionizing the world of web. There are brilliant hardware

devices, amazing software; the need of hour is to bring together the two and

kick-start a new era of IoT that is Internet of Things.

In this project, a hardware device called Myo Gesture control Armband is

integrated with a pre-existing application on the windows system, a special

application for android mobile devices, and with yet another technology called

Arduino.

In the integration of Myo Gesture control Armband with the windows system a

pre-existing application, camera is used. The Myo is integrated with the camera

so that the user do not have to touch the system to click his photograph, make a

video or view the gallery. All will be done by gesturing the arm in which Myo

is worn.

For android, a special general utility mobile application is developed in which

the user can use his selfie stick to click his photograph without setting any

timer. Also, it has a feature by which the user can feel all the notification of his

phone on his arm and can act accordingly.

In the final module, the Myo gesture control armband is integrated with

Arduino through an android mobile application. This has been done to solve the

purpose of home automation. Through the gesture of a hand having Myo, The

electric appliances like, a tube light, a fan, etc. of a room can be controlled.

30

CONCLUSION

Through this report it is concluded that the hardware device Myo Gesture

Control Armband has been studied properly. It has been integrated with a

windows system, an android mobile device and with Arduino in order to

achieve following targets

Control the predefined applications for Windows application with

keyboard mapping of MYO.

Handle calls and read phone notification with a gesture of MYO.

Control the various home appliances using the armband.

On an overall review MYO has turned to be a great device with so much

potential to exist in the market but still it has some shortcomings which is

needed to be overcome for better performance.

31

FUTURE SCOPE

• Home Automation

• Gaming

• Blind Teaching

• E-Learning

• Motion Detection

32

REFRENCES

[1] Dr. Ovidiu Vermesan and Dr. Peter Friess , Internet of Things: Converging

Technologies for Smart Environments and Integrated Ecosystems. River Publishers

2013, pp. 20-100

[2] Mithileysh Sathiyanarayanan, Tobias Mulling and Bushra Nazir, Controlling a

Robot Using a Wearable Device (MYO), School of Computing, Engineering and

Mathematics, UK, 2015, pp. 1-3

[3] Drew Prindle, Myo Gesture Control Armband Review, Digital Trends, 2015

[4] Josie Hughes, Jize Yan and Kenichi Soga, Development of Wireless Sensor

Network using Bluetooth Low Energy (BLE) for Construction Noise Monitoring,

June 15, pp.1394-1400

33

APPENDIX

9.1 Lua Script

scriptId = 'kanika.myoscript.camera'

scriptTitle = "Camera Connector"

scriptDetailsUrl = ""

temp=0

function onForegroundWindowChange(app, title)

myo.debug("onForegroundWindowChange: " .. app .. ", " .. title)

if(title=="Camera") then

temp =0

end

34

if(title=="Photos") then

temp=1

end

return platform == "Windows" and (app == "ApplicationFrameHost.exe" or

title == "Camera")

end

function activeAppName()

return "Camera"

end

function performAction()

myo.keyboard("space", "press")

end

function switchTab()

myo.keyboard("tab", "press")

end

function cancelFocus()

myo.keyboard("escape","press")

end

function switchOptions()

myo.keyboard("up_arrow","press")

end

35

function nextPhoto()

myo.keyboard("right_arrow","press")

end

function previousPhoto()

myo.keyboard("left_arrow","press")

end

function closePhotos()

myo.keyboard("alt+f4","press")

end

function conditionallySwapWave(pose)

if myo.getArm() == "left" then

if pose == "waveIn" then

pose = "waveOut"

elseif pose == "waveOut" then

pose = "waveIn"

end

end

return pose

end

36

function onPoseEdge(pose, edge)

myo.debug("onPoseEdge: " .. pose .. ", " .. edge)

if(temp==0) then

if (pose == "fist") then

if (edge == "on") then

performAction()

myo.setLockingPolicy("standard")

end

if edge == "off" then

myo.unlock("timed")

end

end

if (pose == "waveOut") then

if (edge == "on") then

switchTab()

myo.setLockingPolicy("none")

end

if edge == "off" then

myo.unlock("timed")

end

37

end

if (pose == "waveIn") then

if (edge == "on") then

switchOptions()

myo.setLockingPolicy("none")

end

if edge == "off" then

myo.unlock("timed")

end

end

if (pose == "fingersSpread") then

if (edge == "on") then

cancelFocus()

myo.setLockingPolicy("standard")

end

if edge == "off" then

myo.unlock("timed")

end

end

end

38

if(temp==1) then

if (pose == "fist") then

if (edge == "on") then

performAction()

myo.setLockingPolicy("standard")

end

if edge == "off" then

myo.unlock("timed")

end

end

if (pose == "waveOut") then

if (edge == "on") then

nextPhoto();

myo.setLockingPolicy("none")

end

if edge == "off" then

myo.unlock("timed")

end

end

39

if (pose == "waveIn") then

if (edge == "on") then

previousPhoto();

myo.setLockingPolicy("none")

end

if edge == "off" then

myo.unlock("timed")

end

end

if (pose == "fingersSpread") then

if (edge == "on") then

closePhotos()

myo.setLockingPolicy("standard")

end

if edge == "off" then

myo.unlock("timed")

end

end

end

end

40

9.2 Code snippet for background services that handle various gestures

private DeviceListener mListener = new AbstractDeviceListener() {

@Override

public void onConnect(Myo myo, long timestamp) {

Log.e("myo

connected",""+Hub.getInstance().getConnectedDevices().get(0).getName());

eventBus.post(new String("Connected"));

}

@Override

public void onDisconnect(Myo myo, long timestamp) {

eventBus.post(new String("Disconnected"));

}

// onPose() is called whenever the Myo detects that the person wearing it has

changed their pose, for example,

// making a fist, or not making a fist anymore.

@Override

public void onPose(Myo myo, long timestamp, Pose pose) {

// Show the name of the pose in a toast.

switch (pose) {

case UNKNOWN:

break;

case REST:

case DOUBLE_TAP:

switch (myo.getArm()) {

case LEFT:

break;

case RIGHT:

break;

41

}

break;

case FIST:

break;

case WAVE_IN:

break;

case WAVE_OUT:

break;

case FINGERS_SPREAD:

break;

}

if (pose != Pose.UNKNOWN && pose != Pose.REST) {

// Tell the Myo to stay unlocked until told otherwise. We do that here so you

can

// hold the poses without the Myo becoming locked.

myo.unlock(Myo.UnlockType.HOLD);

// Notify the Myo that the pose has resulted in an action, in this case changing

// the text on the screen. The Myo will vibrate.

myo.notifyUserAction();

} else {

// Tell the Myo to stay unlocked only for a short period. This allows the Myo

to

// stay unlocked while poses are being performed, but lock after inactivity.

myo.unlock(Myo.UnlockType.TIMED);

}

EventBus eventBus = EventBus.getDefault();

eventBus.post(pose);

Log.e("posting event",""+pose);

}

@Override

public void onArmSync(Myo myo, long timestamp, Arm arm, XDirection

42

xDirection) {

super.onArmSync(myo, timestamp, arm, xDirection);

eventBus.post(new String("Synced"));

}

@Override

public void onArmUnsync(Myo myo, long timestamp) {

super.onArmUnsync(myo, timestamp);

eventBus.post(new String("Unsynced"));

}

@Override

public void onUnlock(Myo myo, long timestamp) {

super.onUnlock(myo, timestamp);

eventBus.post(new String("Unlocked"));

}

@Override

public void onLock(Myo myo, long timestamp) {

super.onLock(myo, timestamp);

eventBus.post(new String("Locked"));

}

};

9.3 Code snippet for call receiver

public class IncomingCallReceiver extends BroadcastReceiver {

Context context;

43

private int mInterval = 4000; // 5 seconds by default, can be changed later

private Handler mHandler;

private AudioManager myAudioManager;

public int mod;

public static boolean isRinging = false;

public static Intent intent ;

@Override

public void onReceive(Context context, Intent intent){

try{

String state = intent.getStringExtra(TelephonyManager.EXTRA_STATE);

mHandler =new Handler();

myAudioManager =

(AudioManager)context.getSystemService(Context.AUDIO_SERVICE);

mod=myAudioManager.getRingerMode();

this.intent = intent;

if(state.equals(TelephonyManager.EXTRA_STATE_RINGING)){

Toast.makeText(context, "Phone Is Ringing",

Toast.LENGTH_LONG).show();

startRepeatingTask();

myAudioManager.setRingerMode(AudioManager.RINGER_MODE_SILENT);

isRinging = true;

Log.e("ringing","true");

}

if(state.equals(TelephonyManager.EXTRA_STATE_OFFHOOK)){

Toast.makeText(context, "Call Recieved", Toast.LENGTH_LONG).show();

stopRepeatingTask();

myAudioManager.setRingerMode(AudioManager.RINGER_MODE_VIBRATE);

isRinging = false;

Log.e("ringin","false");

}

44

if (state.equals(TelephonyManager.EXTRA_STATE_IDLE)){

Toast.makeText(context, "Phone Is Idle", Toast.LENGTH_LONG).show();

stopRepeatingTask();

myAudioManager.setRingerMode(AudioManager.RINGER_MODE_VIBRATE);

isRinging = false;

Log.e("ringin","false");

}

}

catch(Exception e){e.printStackTrace();}

}

Runnable mStatusChecker = new Runnable() {

@Override

public void run() {

if(Hub.getInstance().getConnectedDevices().size()>0)

{

if(isRinging){

Hub.getInstance().getConnectedDevices().get(0).vibrate(Myo.VibrationType.LONG);

Log.e("vibration","sent");}

}

mHandler.postDelayed(mStatusChecker, mInterval);

}

};

void startRepeatingTask() {

mStatusChecker.run();

}

void stopRepeatingTask() {

mHandler.removeCallbacks(mStatusChecker);

45

}

}

9.4 Code Snippet for usb serial connection

private SerialInputOutputManager mSerialIoManager;

private final SerialInputOutputManager.Listener mListener =

new SerialInputOutputManager.Listener() {

@Override

public void onRunError(Exception e) {

Log.d(TAG, "Runner stopped.");

}

@Override

public void onNewData(final byte[] data) {

SerialConsoleActivity.this.runOnUiThread(new Runnable() {

@Override

public void run() {

SerialConsoleActivity.this.updateReceivedData(data);

}

});

}

};

@Override

protected void onResume() {

super.onResume();

Log.d(TAG, "Resumed, port=" + sPort);

if (sPort == null) {

mTitleTextView.setText("No serial device.");

} else {

46

final UsbManager usbManager = (UsbManager)

getSystemService(Context.USB_SERVICE);

UsbDeviceConnection connection =

usbManager.openDevice(sPort.getDriver().getDevice());

if (connection == null) {

mTitleTextView.setText("Opening device failed");

return;

}

try {

sPort.open(connection);

sPort.setParameters(115200, 8, UsbSerialPort.STOPBITS_1,

UsbSerialPort.PARITY_NONE);

} catch (IOException e) {

Log.e(TAG, "Error setting up device: " + e.getMessage(), e);

mTitleTextView.setText("Error opening device: " + e.getMessage());

try {

sPort.close();

} catch (IOException e2) {

// Ignore.

}

sPort = null;

return;

}

mTitleTextView.setText("Serial device: " +

sPort.getClass().getSimpleName());

}

onDeviceStateChange();

}

47

48