a wearable hybrid haptic feedback stimulation device for...
TRANSCRIPT
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 104
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
Abstract— A sense of touch restored by upper limb prostheses
can greatly help patients with upper limb amputation to perform
activities of daily living. In this work, a novel hybrid haptic
feedback stimulation system for upper limb prostheses has been
designed, developed and evaluated. The wearable device consists
of one servomotor and two vibration motors, which function as a
pressure feedback display and a vibration feedback display,
respectively. The pressure display conveys the sensing of the
contact pressure and its strength level, while the vibration display
gives an indication about the continuity of the contact pressure. An
evaluation on sensation and response has been conducted with
healthy subjects. The results showed that the subjects can identify
the touch, the start of touch, and the end of touch accurately. In
particular, 96 %, 95 % and 88 % of the healthy subjects are able
to distinguish the grasp, the pressure level, and the slipping
objects, respectively. In addition, the usability tests demonstrated
that the system is easy to setup and comfortable to use. The device
is able to convey real-time haptic feedback to the amputees without
the risk of surgery, brain confusion and intensive pre-training.
This solution contributes to making the haptic feedback
stimulation a common functionality in upper limb prostheses in
the near future.
Index Term— Feeling recovering, Haptic feedback stimulation
system, Mechanotactile stimulation, Tactile glove, Upper limb
prostheses, Vibrotactile stimulation.
I. INTRODUCTION
The human hand can be regarded as a multi-degree of freedom
(DOF) system, with massive number of sensors, actuators,
tendons, and a complex control strategy which can perform
multi-tasks in daily life, such as grasping, manipulating objects,
sensing, interacting with the environment, and using gestures to
support speech and express emotions. The loss of human hand
causes severe physical and mental illness [1]. Prostheses have
emerged to help with such physical loss and partially restore the
lost functionality. The upper limb prostheses help the patients
with upper limb amputation to touch and grasp objects, but
compensating the loss of sensation during the use of prosthetic
hands is still in progress. There have been numerous efforts to
develop and equip the prostheses with the haptic feedback
stimulation system, so that the amputees can interact with the
surrounding and restore the sensation as natural as possible.
The haptic feedback stimulation system can be divided into two
main techniques: invasive and non-invasive, depending on the
methods used to stimulate the patient’s peripheral nerves in
order to restore the somatosensory perception. The invasive
technique depends on the surgical intervention and surgical
access to the nerves of the amputees [2, 3], while the non-
invasive technique relies on exciting the patient's nerves
externally by fixing haptic wearable devices on the amputees'
residual parts [4-6]. More specifically, the non-invasive
feedback stimulation system – the priority of this work, can be
further classified into six stimulation systems, depending on
how to stimulate the patient’s skin and provide the information
of the sensory system to the amputee’s brain. The five feedback
stimulation displays are: pressure feedback stimulation system
[7-9], vibration feedback stimulation system [10-13], skin
stretch feedback stimulation system [4, 6, 14-16], squeeze
feedback stimulation system [5, 17, 18], electro feedback
stimulation system [19-23], and thermal feedback stimulation
system [24-30].
Numerous works have been done in order to identify the
most suitable type of haptic feedback display. In general,
comparison between two types displays, like comparison the
rotating and the linear vibration actuators or comparison the
vibration and the skin-stretch wearable devices, were
investigated to convey the sensory information to the amputees'
brain with a high response and good accuracy [31, 32]. On other
hand, several studies revealed the benefits of combining two or
more haptic feedback displays as a hybrid stimulation system
in order to increase the performance of the haptic wearable
device [33-37].
Several issues related to equipping the myoelectric
prosthesis with the haptic feedback stimulation system were
investigated in literature. Firstly, issues related to the
comfortability when using the haptic prosthetic hand, like the
heavyweight [31], the high noise of actuators [38], and absence
of sensation through the prostheses [39] were studied. The
increasing in energy consumption due to using the auxiliary
equipment of the haptic feedback stimulation system was
diagnosed as well [11, 37]. Moreover, issues concerned with the
ability of patient’s brain to recognize multi information
delivered from the sensory system at the same time [12]. For
example, when the sensory system of the prosthetic hand has
pressure, vibration, and temperature sensors, accordingly the
patient’s brain has to distinguish the contact pressure, surface
texture, and the surface temperature at the same time, which
lead to reduce the brain’s recognition accuracy. Finally, issues
regarded to the prostheses design itself.
Mohammed Najeh Nemah1,2, Saif Salih Khaleel3, Omar Hammad Aldulaymi1, Cheng Yee Low1*, Pauline Ong1, Balasem Abdulameer Jabbar1,2
A Wearable Hybrid Haptic Feedback
Stimulation Device for Upper Limb Prostheses
1Faculty of Mechanical and Manufacturing Engineering, University Tun
Hussein Onn Malaysia, 86400, Parit Raja, Batu Pahat, Johor, Malaysia
([email protected]). 2Engineering Technical College-Najaf, Al-Furat Al-Awsat Technical
University, 54001, Najaf, Iraq ([email protected]). 3Biomedical Engineering Department, Al-Mustaqbal University College,
51001 Hillah, Babil, Iraq ([email protected] ).
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 105
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
It is extremely difficult to design and product haptic upper limb
prostheses that capable to perform whole the users' life
activities comparable to the human real hand. For instance, the
ability of the users to detect and avoid the painful stimulus [9].
In general, it can be concluded that the two main disadvantages
of haptic feedback stimulation system are: (i) patient’s brain
might confuse due to the massive amount of data provided from
the sensory system during the operating of hand prostheses, and
(ii) the need of long hours of pre-training on how to recognize
the sensory information [40].
The main objective of this study is to design a hybrid haptic
pressure-vibration feedback stimulation system (HHPVFSS),
which has the ability to assist the patients of the upper limb
amputation to perceive multi-information about their
surroundings, at the same time, without confusing or requiring
a pre-training. HHPVFSS includes two main parts: the tactile
sensory system and the haptic feedback stimulation system. The
tactile sensory system aims to gather the perceptual information
about the touch, grasp, and the slippage by utilizing the
piezoresistive pressure sensors distributing over the hand. The
feedback stimulation system (FSS) is responsible for the
human-machine interface, by conveying the perceptional
information to the patient's brain through the hybrid wearable
device, which consists of the pressure feedback display and the
vibration feedback display. To the best of our knowledge, this
is the first endeavor to combine haptic pressure and vibration
feedback displays for upper limb prostheses. Such combination
allows the stimulators to operate in different directions and
deliver the tactile sensory information to the users in a quick
manner.
II. DESIGN CONCEPT OF HAPTIC FEEDBACK STIMULATION SYSTEM
In general, the HHPVFSS of upper limb prostheses can be
divided into three main parts: tactile sensory system, computer
system and haptic feedback stimulator system. The tactile
sensory system is responsible for collecting the information
from the environment and converting the information into
measurable data by utilizing different types of sensors. The
computer system is accountable for processing and analyzing
the sensory data, and subsequently, manipulating the comments
signals to control the feedback stimulators to excite the
amputee's brain. Finally, the haptic feedback stimulator system
aims to stimulate the amputees’ nervous system by exciting the
skin of their residual parts, for instance, the forearm, upper
limb, or any other parts of the body.
A. Design Concept Of The Tactile Pressure Sensory System
The main goal of this study is to design a tactile pressure
glove with the ability to detect the contact pressure. The
developed haptic system should have the ability to detect the
contact pressure with six pressure sensors and provide the
measured information to the patient’s brain by utilizing a single
hybrid pressure-vibration feedback stimulation device.
Therefore, the computer system should be programmed in an
effective way in order to select the largest pressure signal
among the six signals of the pressure sensors and transfer it to
the single actuator feedback stimulation device, which can be
fixed on the residual limbs of the amputees.
In order to develop a functional tactile prosthetic hand, the
tactile sensor has to be flexible enough to coat the curving
surfaces of the prosthetic hand [41]. Also, the tactile
force/pressure sensor should be rigid and sensitive enough to
measure the static and the dynamic forces applied on the
prosthetic hand. The piezoresistive force is then utilized to
detect the movement of the slipping objects [42]. In general, the
piezoresistive force sensor is fabricated from a semiconductor
material, which has the ability to change its electrical resistance
when an effective force is applied directly over it.
Based on the above acquaintance, the quantum tunnelling
composites (QTC) with 10 mm diameter, piezoresistive
pressure sensor was chosen to develop a haptic feedback
stimulation system because it has the ability to detect the touch,
grasp, and the object slippage at the same time, in order to cover
the critical points of the touch on the prosthetic hand like the
palm zone and five fingertips. Therefore, five QTC sensors
were fixed on each fingertip and an extra sensor was mounted
on the hand’s palm to increase the chance of capturing the force
affecting the hand, as shown in Fig.1.
Fig. 1. Design conception of a pressure sensory system
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 106
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
The signals from the pressure sensors were transferred to the
computer system by utilizing the Arduino Mega 2560
microcontroller, as shown in Fig.2. After that, the six pressure
signals obtained from the QTC force sensors were compared.
The largest instant signal among all was selected as the output
signal of the sensory system. Based on the output sensory
signal, the computer system controlled the feedback stimulation
actuators by providing special excitation with different effects
for both the pressure and the vibration feedback stimulation
systems.
Fig. 2. Design concept of a HHPVFSS.
B. Design concept of hybrid pressure-vibration wearable feedback device
On the other hand, a pair of 10 mm diameter, circular
vibration motors and a single servo motor were used
simultaneously, in order to provide pressure and vibration
stimulation to the skin of the amputee’s upper arm by exciting
his nervous system.
The servomotor of the pressure feedback stimulation system
was rotated from 0 to 35 degrees when the sensory system
signal was increased from 0 to 20 N. This was to notify the
amputees that the contact pressure between the prosthetic hand
and the grasping object was increased constantly.
The vibration motors were excited with a special periodical
signal, varying from 0 to 3 V for every 0.2 sec, according to any
value of the applied force. This was to inform the amputees that
the contact pressure was still active and the touching or
grasping action was continued. The vibration motors were
activated when the sensory signal exceeded 0.5 V. This was to
prevent the activation of the vibration feedback display when
the phantom signals were generated by the pressure sensors,
contributed by the hysteresis behaviors at no load case. Finally,
the Arduino Mega 2560 microcontroller was used to transfer
the excitation signals to the feedback stimulation actuators, as
presented in Fig.2.
III. FABRICATION OF HAPTIC FEEDBACK STIMULATION SYSTEM
A. Fabrication of Tactile Pressure Sensory System
A plastic glove equipped with spot pressure sensors was
designed as the sensory system of the HHPVFSS. The design concept, fabrication, and the evaluation tests of the tactile
pressure glove have been studied in previous own work [43]. In
general, the tactile glove fabricated from a plastic glove was
shielded with rigid foundations located under the pressure
sensors, to provide sufficient space for installing the pressure
sensors and increase its performance.
The rigid shield was designed using the Solidwork 2018
program and printed by a 3D printer of type Raise-3D-N2-Plus
[44], using Acrylonitrile Butadiene Styrene (ABS) material
[45]. Six QTC pressure sensors of SP200-10 series with 10 mm
diameter and 0.1 N to 20 N operating force from Peratech [46] were distributed over the hand. Five sensors were mounted on
each fingertip and one sensor was fixed on the palm. Such
arrangement of sensors was to cover all critical areas of the
hand and increase the probability to detect the contact pressure
while touching the surfaces or grasping the objects.
Based on this type of fabrication, the benefits obtained are: (i)
the tactile pressure glove can be equipped easily with the
prosthetic hand without obstructing the movement of the
fingers’ joints because the glove is made from elastic material;
(ii) the rigid shields of fingertips and the palm enables the base
of the pressure sensors to be stiff enough to work with high
accuracy; and (iii) the glove is appropriate for various sizes and types of the prosthetic hands due to its extensibility behavior.
The designed HHPVFSS has only one feedback stimulator,
but this singular stimulator has to convey the data of six
pressure sensors to the patient’s brain. To solve the problem,
the tactile sensory system was programmed to output the largest
instant signal from all the pressure signals. If the time
dependent vectors P⃑⃑ (t), R⃑⃑ (t), M⃑⃑⃑ (t), I (t), T⃑⃑ (t), and Pa⃑⃑⃑⃑ (t) represent the pressure sensors signals of the pinky fingertip,
ring fingertip, middle fingertip, index fingertip, thumb
fingertip, and the palm, respectively, the largest signal output
from the tactile sensory system 𝑉𝑂𝑢𝑡⃑⃑ ⃑⃑ ⃑⃑⃑⃑ (𝑡) can be calculated as:
VOut⃑⃑⃑⃑ ⃑⃑⃑⃑ (t) = Max[P⃑⃑ (t), R⃑⃑ (t), M⃑⃑⃑ (t), I (t), T⃑⃑ (t), Pa⃑⃑⃑⃑ (t)] (1)
The InMoov prosthetic arm was used to evaluate the proposed
pressure sensory system, as shown in Fig. 3. The design and the
necessary files of the InMoov prosthetic arm are open sources [47]. The prosthetic arm was built using 3D printer,
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 107
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
servomotors, artificial tendon, and Arduino Mega 2560
microcontroller. After that, the tactile pressure glove was
integrated to the InMoov prosthetic arm.
When the pressure sensors are excited by an external load or
pressure; the pressure sensors will measure this excitation and
present it as an analog signal in voltage. Therefore, the calibration of the pressure sensors is highly recommended to
calculate the relationship between the applied load or pressure
and the output of the pressure sensors. In addition, the
calibration is performed to increase the measuring accuracy and
minimize the error. The output signal of the sensor is then
compared to an accurate static load, and the relationship
between the measured and the accurate value is determined.
For this reason, a load stand device was designed and printed
using a 3D printer, as described in Fig. 4. The device is mounted
on the tactile prosthetic hand and the load is applied gradually,
with 0.5 kg for each increment. Then, the sensor output voltage
is recorded against the increasing of the static load. The calibration process is repeated for each of the six pressure
sensors individually.
The result of the calibration process is described in Fig. 5. The
horizontal and the vertical axes are the applied load and the
output measuring voltage, respectively. In order to model the
relationship between the independent variable (applied load)
and the dependent variable (pressure sensor signal), the 6-th
order polynomial was chosen as the fitting model after
numerous attempts. The obtained fitting model is given as:
FL = P1 × l6 + P2 × l
5 + P3 × l4 + P4 × l
3 + 4.923 × l2 +P5 × l + P6 (2)
Where: l represents the applied load (kg) and FL is a load fitting function. The obtained R-squared of the fitting model is
0.9959, indicating that the data are well explained by the fitting
model. While the values P1−5 are the fitting equation constants, which equals to P1 = 0.01962, P2 = −0.2925, P3 = 1.67, P4 = −4.447, P5 = 4.923, and P6 = −0.01707.
Fig. 3. Equipping the InMoov prosthetic arm with the tactile pressure
sensory system.
Fig. 4. Calibrating the pressure sensory system.
B. Fabrication of hybrid pressure-vibration feedback wearable device
The main aim of the HHPVFSS is to stimulate the amputees’
residual upper limb and convey the sensation of the touch, start
of touch, end of touch, pressure level, grasp, and slippage to the patient’s brain. Accordingly, a hybrid haptic pressure-vibration
feedback stimulator was designed to augment the haptic
sensation and enable the amputees’ brain to recognize the
massive amount of data provided by the pressure sensory
system. The wearable device consists of one SG-90 servomotor
[48] for the pressure stimulation and two linear resonant
actuator (LRA) vibration motors [49] for the vibration
stimulation.
Fig. 5. The performance of the pressure sensors vs. the applied load.
The three actuators were fixed on a curvature wearable case
of 70 cm length and 8 mm thickness, as shown in Fig.6. The
curvature case was designed using Solidwork 2018 and printed
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 108
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
by 3D printer using ABS material [45]. Since the two vibration
motors can work independently and provide the vibration
stimulation directly to the arm, therefore, it was fixed at the
inner side of the curvature wearable case in direct contact with
the patient’s skin. On the other hand, a circular piston of 15 mm
diameter and piston’s arm of 33 mm length have been added to the servomotor to deliver the pressure stimulation.
IV. INTERFACING THE HAPTIC WEARABLE DEVICE WITH THE COMPUTER SYSTEM
The main function of the computer system is to manipulate
and control the actuators of HHPVFSS according to the output
signal obtained from the pressure sensory system. Two different
types of signals were generated by the pressure sensory system.
These signals would be utilized to excite the pressure servomotor and vibration motors of the wearable device. The
pressure actuator would apply a pressing action on the patient’s
upper arm, which is proportional to the largest signal value of
the pressure sensors. Therefore, the value generated by the
pressure sensor within the range of 0 to 5 V will be converted
to a rotational excitation signal varying from 0 to 35 degrees, in
order to control the servomotor and satisfy the pressure
stimulation.
On the other hand, QTC sensor has an unstable behavior at
the absence of applying load [43]. Hence, the vibration motors
only starts working if the pressure signal is equal or more than 0.5 V. Then, the vibration actuators are excited by a specific
generated signal varying from 0 to 3 V with a suitable frequency
to provide a sensible and comfortable stimulation to the
amputees.
Fig. 6. The fabrication of HHPVFSS: a) rear view, and b) inside view.
The Arduino Mega 2560 microcontroller was used to connect
the SG-90 servomotor and the two LRA vibration motors with
the computer system, as described in Fig. 7. The Arduino
microcontroller was equipped with an external DC power
supply to cover the electrical loads required for operating the
pressure and vibration motors.
Fig. 7. Connecting the actuators of the haptic wearable device with the
Arduino microcontroller.
V. THE EVALUATION AND FUNCTIONALITY TESTS
Two functionality tests, specifically, the increasing load test
and slippage detection test, were conducted to evaluate the
functionality of the developed HHPVFSS. Additionally, the
evaluation tests on the system’s response, analysis of the
wearable haptic device behavior and its actuators due to varying the environment’s parameter measured by the tactile pressure
sensory system, were performed as well.
The experimental evaluation of the increasing load test has
been set up by using the load stand as described in Fig. 4. A
static weight was applied directly to the pressure sensor
mounted on the fingertip. The applied mass was increased from
0 to 2 kg by adding 200 g for each increment. The purpose of
this evaluation test is to calibrate the response of the servomotor
and the vibration motors due to the increase of the pressure
sensor signal.
The experimental evaluation of the slippage detection test has been set up by utilizing a cylindrical pipe equipped with a steel
hook at its bottom side. The cylindrical pipe was grasped and
held by the tactile prosthetic hand. Subsequently, various
weights were hanged to the hook of the cylindrical pipe, as
shown in Fig. 8. The first aim of this test is to evaluate the
performance of the stimulators when the load is increased from
0 to 5.5 kg by adding 0.5 kg for each increment. The second
aim is to validate the functionality of the HHPVFSS to detect
the slipping objects.
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 109
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
Fig. 8. Evaluation of slippage detection experiment.
VI. EXPERIMENTAL SETUP AND PROCEDURE
A total of 25 healthy volunteers were involved in this study
(mean age (SD) 27.68 ± 3.92 years). The healthy volunteers were used instead of the amputees’ volunteers for two main
reasons. The first reason is, exciting the human’s nervous
system is the main goal of the experimental testing, and hence,
it is not very important to use amputees’ volunteers. The second
reason is the difficulty to get amputees at any time or place. The
participants were right-hand dominant and did not suffer from any cognitive impairment that could affect their performance
during the tests. Volunteers were seated comfortably against the
table. They wore the haptic wearable device on their upper right
arm, as shown in Fig. 9. The eyes mask was used to ensure that
the volunteers only rely on the haptic information provided by
the haptic wearable device, since the participants have no visual
information of their surrounding during the test. Additional
earmuffs were worn to prevent an auditive detection.
The measured outcome in all the experimental examinations
is the stimuli identification rate (SIR), i.e., the rate that the
volunteers could correctly identify the tactile information by only utilizing the HHPVFSS.
In the first test, the examiner applied random forces on the
pressure sensors of the tactile sensory system through touching
the fingertip of the tactile prosthetic hand, as shown in Fig. 10.
The volunteers were asked if they could discriminate the touch,
start of touch, and end of touch. This is to evaluate the
effectiveness of the haptic wearable device in delivering
sensory information to the patients’ brain by stimulating their
nervous system. Evaluating the ability of the volunteers to
recognize the grasping objects was conducted in the second test.
The volunteers were asked to detect the grasping forces when
the prosthetic hand grasped a small ball or made a handshake
with the examiner, as presented in Fig. 11. This was to examine the functionality of the HHPVFSS to detect the contact pressure
when grasping an object.
Fig. 9. Experimental setup: the volunteer wore HHPVFSS with his right
hand.
Fig. 10. Touch detection test.
In the third test, the loads applied on the fingertip of the tactile
prosthetic hand were increased slowly. The volunteers have to
identify whether they were able to detect the changing force.
This test attempted to measure the effectiveness of the haptic wearable device in delivering a sense of increasing pressure to
the patient's brain.
Finally, the slippage detecting test was performed by making
the prosthetic arm to grasp and hold the cylindrical pipe, while
the hanging load was increased gradually, until the pipe slipped
down from hand, as shown in Figure 4.10. The main goal of this
test was to identify the ability of the HHPVFSS to convey the
information about the slipping objects to the users of the haptic
upper limb prostheses.
The experimenter recorded the volunteers' answers for all the experimental tests as follows: (i) yes, if the volunteers were able
to recognize the type and the duration of the excitation during
the test, (ii) no, if the volunteers were confused and unable to
analyze the haptic information. The comfortability of wearing
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 110
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
and using the haptic wearable device evaluated by the
volunteers was taken into account as well.
Fig. 11. Gasping detection test when the prosthetic arm: a) grasps a ball,
and b) shake hands with the experimenter.
VII. EXPERIMENT RESULTS AND DISCUSSION
A. Experiment of The Hybrid Pressure-Vibration Wearable Feedback Device
For the increasing load test, Fig. 12.a presents the linear
relationship between the rotational movement of the
servomotor and the largest instant sensory signal. The rotational
movement of the servomotor was varied from 0 to 35 degrees
when the pressure signal of the sensory system was increased
from 0 to 5 V, i.e. the load was increased from 0 to 20 N. Based
on the servomotor’s maximum rotational movement (35
degree) and the length of servomotor’s piston arm, the vertical
displacement of the servomotor’s piston was calculated, where
15.6 mm was obtained. This implied that the pressure feedback
display of the HHPVFSS was capable to deform the skin of the user’s upper arm with 15.6 mm vertical displacement. This
value of displacement is suitable to excite patient’s nervous
system and inform the patient’s brain about the increasing load.
This is due to, in general, this value is the highest value that can
be applied to reach the arm bone.
Fig. 12. The relationship between the wearable device’s actuators and the
largest instant sensory signal.
For the vibration feedback display of the HHPVFSS, a
repeatable pulses signal with 3 V maximum amplitude, 0.2 sec
period time, and a pulse width equal to 50 % of the period time was designed to drive the two vibration motors, as shown in
Fig. 13. The special pulses vibration motor signal was designed
based on several stimulation trials, in order to obtain the best
feedback stimulation from the vibration feedback display
without any interference with the noise frequency of the
servomotor.
The special pulsing signal was applied to drive the two
vibration motors at the instant when the contact with the
surfaces or objects took place. Therefore, the vibration motors
were excited by only one type of stimulation signal in all
operation, i.e. the vibration motors never got affected with the
increasing load. In addition, the vibration motors were excited at the instant when the pressure sensors signal exceeded 0.5 V,
in order to avoid the undesirable hysteresis phenomenon of the
piezoresistive pressure sensors that were distributed over the
tactile glove, as described in Fig. 12.b and Fig. 12.c.
Fig. 13. The vibration motor input signal during 1 sec.
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 111
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
On the other hand, Fig. 14 shows the responses of the large
instant pressure sensors signal, the servomotor’s rotational
movement, the special periodical pulsing signal that input to the
vibration feedback display, and the vibration output signal
during increasing the hanging mass from 0 to 5.5 kg by adding
0.5 kg for each increment of every 10 sec. The results of the slippage detection test showed that the rotational movement of
the servomotor was increased gradually when the load was
increased systematically. In addition, the controlling signal of
the vibration motors was generated for all operating period, but
it was only being allowed to pass the signal to the vibration
motors when the signal of the tactile sensory system exceeded
0.5 V.
Furthermore, the test results corroborated the functionality of
the HHPVFSS to help the amputees to recognize the slipping
objects by utilizing the haptic upper limb prostheses. It was
found that after the hanging mass reached 5.5 kg, the volunteer
was unable to hold the pipe. It can be noted that, when the load was decreased at the operation time of 110 secs, the output
signals of the servomotor and the two vibration motors were
rapidly reduced to zero, indicating there was no contact
between the prosthetic hand and the slipped object.
Fig. 14. The responses of the sensors and stimulators when the hanging
weight increased gradually.
B. User Experience Evaluation for the Hybrid Pressure-Vibration Feedback Stimulation System
The SIR of 25 healthy volunteers, which examined the ability
of the HHPVFSS to convey the tactile information to the brain
is described in Fig. 15. The final results were completely based
on the 100 answers taken from all the volunteers for each
experiment test (25 volunteers and four repetitions for each
test). The results showed that all volunteers were able to
recognize the touch, start of touch, and end of touch correctly
with 100 % accuracy. These results gave a positive impression of the effectiveness of the haptic wearable device in transferring
the tactile data.
In particular, 96 %, 95 % and 88 % of the volunteers were
able to detect the grasping objects, the pressure level, and the
slippage, respectively. The relatively low recognition
percentage of the slippage examination could be related to the
design of the tactile glove itself. The tactile glove does not have
a real sliding sensor to detect the slippage. It only depends on
the combination between the pressure sensors and the responses
of the volunteers’ nervous system.
Fig. 15. Stimuli identification rate of the able-body volunteers.
A comparison study between the SIR results of the current
work and eight previous studies was made, as presented in Table 1. The SIR results during touch, grasp, pressure level, and
the slippage examinations were considered. The selected
previous studies were as similar as possible to this work, in
which a pressure sensory system, depending on spot pressure
sensors was developed in the respective works. In addition,
previous studies used different types of feedback stimulation
system, like pressure, vibration, electro, squeeze, and skin
stretch to compare the functionality of the designed wearable
device with all types of existing feedback stimulation systems.
On the other hand, Table 1 summarizes too the information
about the installing position of the previous haptic feedback
displays, type of subjects, and the determination about whether the previous experiments were performed with or without a pre-
training.
The SIR results indicated that the current work recorded the
highest percentage in all the experiment examinations.
Moreover, it should be noted that as compared with previous
studies, no initial exercises and pre-training were included in
this work in order to increase the recognition rate of the
volunteers during the experiments.
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 112
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
VIII. CONCLUSION A novel HHPVFSS of the upper limb prostheses was
designed, in order to help the patients of upper limb amputation
to restore the sensation of their surroundings. The wearable
device was designed to stimulate the user’s upper arm skin by
means of utilizing pressure and vibration feedback stimulation systems. A tactile pressure glove was designed from a plastic
material and equipped with five QTC pressure sensors mounted
on each fingertips and an extra sensor was placed on the palm.
The tactile glove was integrated with the InMoov prosthetic arm
to produce a completely tactile system. On the other hand, the
haptic wearable device was fabricated from one SG-90
servomotor and two LRA vibration motors, which work
together to stimulate the patients’ nervous system.
The design of the tactile glove is the first step for designing a
complete haptic feedback stimulation system to enable the
amputees to sense their surrounding while they wear their own
haptic upper limb prostheses. Therefore, the haptic feedback stimulator is designed in order to provide the useful information
about the contact pressure and the slippage to the amputees’
brain with high response, acceptable accuracy, low noise and
power consumption, and without any brain’s confusing.
However, some limitations of the study need to be considered.
1. It is so difficult to find the amputees volunteers with upper limb amputation to perform the evaluation and
the functionality tests.
2. The designed HHPVFSS only has a single haptic stimulator that is capable to work with one sensory
signal, hence, it is very difficult to install a haptic wearable device that has six stimulators on the
patient’s upper arm to work with each sensory signal
individually.
The experimental results of evaluation tests outcomes reveal
that HHPVFSS provided a suitable level of accuracy and an
acceptable response for detecting the contact pressure. In
general, the evaluation tests and the SIR examinations proved
the functionality and the effectiveness of the HHPVFSS to
provide multi-information to the amputee’s brain without
confusing or pre-training. The SIR examinations recorded 100
% percentage recognition accuracy for the touch, start of touch,
and end of touch experimental tests, while about 96 %, 95 %
and 88 % of the participants were able to perceive the grasping
objects, pressure level and the slippage, respectively.
To the best of our knowledge, this is the first design of a
complete non-invasive haptic feedback stimulator combining the pressure and the vibration feedback stimulation system, in
order to work together at the same operation time. The pressure
feedback was used to convey the information about the contact
pressure and its strength level, while the vibration feedback was
programmed to be an auxiliary stimulator to give the user an
impression of continued contact pressure. In future work, the
HHFSS will be improved further to detect the surface texture
and surfaces temperature in addition to the contact pressure
detection.
Acknowledgments
The authors would like to express their gratitude to Research Fund RMC [Vot E15501] from University of Tun Hussein Onn
Malaysia for funding the research work.
List of abbreviations and symbols:
HHFSS : Haptic hybrid feedback stimulation system.
DOF : Degree of freedom.
ABS : Acrylonitrile Butadiene Styrene.
QTC : Quantum tunnelling composites.
LRA : Linear Resonant Actuator.
SIR : Stimuli identification rate.
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 113
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
TABLE I
Comparison between the stimuli identification rate of the current study with the previous works.
REFERENCES [1] K. Ziegler-Graham, E. J. MacKenzie, P. L. Ephraim, T. G. Travison,
and R. Brookmeyer, "Estimating the prevalence of limb loss in the
United States: 2005 to 2050," Archives of physical medicine and
rehabilitation, vol. 89, pp. 422-429, 2008.
[2] E. L. Graczyk, M. A. Schiefer, H. P. Saal, B. P. Delhaye, S. J.
Bensmaia, and D. J. Tyler, "The neural basis of perceived intensity
in natural and artificial touch," Science translational medicine, vol.
8, pp. 362ra142-362ra142, 2016.
[3] M. Schiefer, D. Tan, S. M. Sidek, and D. J. Tyler, "Sensory feedback
by peripheral nerve stimulation improves task performance in
individuals with upper limb loss using a myoelectric prosthesis,"
Journal of neural engineering, vol. 13, p. 016001, 2015.
[4] M. Rossi, M. Bianchi, E. Battaglia, M. G. Catalano, and A. Bicchi,
"Hap-pro: a wearable haptic device for proprioceptive feedback,"
IEEE Transactions on Biomedical Engineering, 2018.
[5] L. Meli, I. Hussain, M. Aurilio, M. Malvezzi, M. O'Malley, and D.
Prattichizzo, "The hBracelet: a wearable haptic device for the
distributed mechanotactile stimulation of the upper limb," IEEE
Robotics and Automation Letters, vol. 3, pp. 2198 - 2205, 2018.
[6] J. P. Clark, S. Y. Kim, and M. K. O’Malley, "The rice haptic rocker:
Altering the perception of skin stretch through mapping and
geometric design," in IEEE Haptics Symposium (HAPTICS), San
Francisco, CA, USA, 2018.
[7] G. Savioz, V. Ruchet, and Y. Perriard, "Study of a miniature
magnetorheological fluid actuator for haptic devices," in Advanced
Intelligent Mechatronics (AIM), 2010 IEEE/ASME International
Conference on, Montreal, ON, Canada, 2010, pp. 1197-1202.
[8] M. O. Culjat, J. Son, R. E. Fan, C. Wottawa, J. W. Bisley, W. S.
Grundfest, et al., "Remote tactile sensing glove-based system," in
Engineering in Medicine and Biology Society (EMBC), 2010 Annual
International Conference of the IEEE, Buenos Aires, Argentina,
2010, pp. 1550-1554.
[9] S. Fukushima, H. Sekiguchi, Y. Saito, W. Iida, T. Nozaki, and K.
Ohnishi, "Artificial Replacement of Human Sensation Using Haptic
Transplant Technology," IEEE Transactions on Industrial
Electronics, vol. 65, pp. 3985-3994, 2018.
[10] M. Nabeel, K. Aqeel, M. N. Ashraf, M. I. Awan, and M. Khurram,
"Vibrotactile stimulation for 3D printed prosthetic hand," in
Robotics and Artificial Intelligence (ICRAI), 2016 2nd International
Conference on, Rawalpindi, Pakistan, 2016, pp. 202-207.
[11] M. D'Alonzo, F. Clemente, and C. Cipriani, "Vibrotactile
stimulation promotes embodiment of an alien hand in amputees with
phantom sensations," IEEE Transactions on Neural Systems and
Rehabilitation Engineering, vol. 23, pp. 450-457, 2015.
[12] M. R. Motamedi, J.-P. Roberge, and V. Duchaine, "The Use of
vibrotactile feedback to restore texture recognition capabilities, and
the effect of subject training," IEEE Transactions on Neural Systems
and Rehabilitation Engineering, vol. 25, pp. 1230-1239, 2017.
[13] J. M. Fontana, R. O’Brien, E. Laciar, L. S. Maglione, and L.
Molisani, "Vibrotactile Stimulation in the Upper-Arm for Restoring
Individual Finger Sensations in Hand Prosthesis," Journal of
Medical and Biological Engineering, vol. 38, pp. 782–789, 2018.
[14] K. Bark, J. Wheeler, G. Lee, J. Savall, and M. Cutkosky, "A
wearable skin stretch device for haptic feedback," in EuroHaptics
conference, 2009 and Symposium on Haptic Interfaces for Virtual
Environment and Teleoperator Systems. World Haptics 2009. Third
Joint, Salt Lake City, UT, USA, 2009, pp. 464-469.
[15] O. Kayhan, A. K. Nennioglu, and E. Samur, "A skin stretch tactor
for sensory substitution of wrist proprioception," in Haptics
Symposium (HAPTICS), 2018 IEEE, San Francisco, CA, USA,
2018, pp. 26-31.
[16] E. Battaglia, J. P. Clark, M. Bianchi, M. G. Catalano, A. Bicchi, and
M. K. O'Malley, "The Rice Haptic Rocker: skin stretch haptic
feedback with the Pisa/IIT SoftHand," in World Haptics Conference
(WHC), 2017 IEEE, Munich, Germany, 2017, pp. 7-12.
[17] D. D. Damian, M. Ludersdorfer, Y. Kim, A. H. Arieta, R. Pfeifer,
and A. M. Okamura, "Wearable haptic device for cutaneous force
and slip speed display," in Robotics and Automation (ICRA), 2012
IEEE International Conference on, Saint Paul, MN, USA, 2012, pp.
1038-1043.
[18] E. Treadway, B. Gillespie, D. Bolger, A. Blank, M. O'Malley, and
A. Davis, "The role of auxiliary and referred haptic feedback in
References
Type of
sensory
system
Type of
Feedback
display
Setup of
Feedback
display
Type of subjects
Experimental
with
pre-training
Stimuli identification rate (SIR)
Touch Grasp pressure
level Slippage
Current
Study Pressure
Hybrid Pressure
and Vibration Upper limb
Healthy
volunteers No 100% 96% 95% 88%
[8] Pressure Vibration Forearm Healthy
volunteers No 99.3% 94.7%
[50] Pressure Vibration Upper limb Healthy
volunteers No 94% 75%
[51] Pressure Vibration Upper limb Healthy
volunteers Yes 93.9% 85%
[34] Pressure Pressure and
Vibration Upper limb
Amputees
volunteers Yes 93% 90%
[19] Pressure Electro Forearm
Healthy and
amputees
volunteers
Yes 90%
[17] Pressure Squeeze Forearm Healthy
volunteers Yes 70.3% 62.2%
[16] Pressure Skin stretch Upper limb Healthy
volunteers Yes 73.3%
[10] Pressure Vibration Forearm
Healthy and
amputees
volunteers
No
Yes
72%
94%
-
International Journal of Mechanical & Mechatronics Engineering IJMME-IJENS Vol:19 No:05 114
190205-3737-IJMME-IJENS © October 2019 IJENS I J E N S
myoelectric control," in World Haptics Conference (WHC), 2015
IEEE, Evanston, IL, USA, 2015, pp. 13-18.
[19] M. Štrbac, M. Belić, M. Isaković, V. Kojić, G. Bijelić, I. Popović,
et al., "Integrated and flexible multichannel interface for
electrotactile stimulation," Journal of neural engineering, vol. 13,
p. 046014, 2016.
[20] M. Franceschi, L. Seminara, S. Dosen, M. Strbac, M. Valle, and D.
Farina, "A system for electrotactile feedback using electronic skin
and flexible matrix electrodes: Experimental evaluation," IEEE
transactions on haptics, vol. 10, pp. 162-172, 2017.
[21] E. I. Germany, E. J. Pino, and P. E. Aqueveque, "Myoelectric
intuitive control and transcutaneous electrical stimulation of the
forearm for vibrotactile sensation feedback applied to a 3D printed
prosthetic hand," in Engineering in Medicine and Biology Society
(EMBC), 2016 IEEE 38th Annual International Conference of the,
Orlando, FL, USA, 2016, pp. 5046-5050.
[22] M. Štrbac, M. Isaković, M. Belić, I. Popović, I. Simanić, D. Farina,
et al., "Short-and Long-Term Learning of Feedforward Control of a
Myoelectric Prosthesis with Sensory Feedback by Amputees," IEEE
Transactions on Neural Systems and Rehabilitation Engineering,
vol. 25, pp. 2133-2145, 2017.
[23] D. Pamungkas and K. Ward, "Tactile sensing system using electro-
tactile feedback," in Automation, Robotics and Applications
(ICARA), 2015 6th International Conference on, Queenstown, New
Zealand, 2015, pp. 295-299.
[24] S. Gallo, C. Son, H. J. Lee, H. Bleuler, and I.-J. Cho, "A flexible
multimodal tactile display for delivering shape and material
information," Sensors and Actuators A: Physical, vol. 236, pp. 180-
189, 2015.
[25] M. Hojatmadani and K. Reed, "Asymmetric Cooling and Heating
Perception," in International Conference on Human Haptic Sensing
and Touch Enabled Computer Applications, 2018, pp. 221-233.
[26] M. Gabardi, D. Leonardis, M. Solazzi, and A. Frisoli, "Development
of a miniaturized thermal module designed for integration in a
wearable haptic device," in Haptics Symposium (HAPTICS), 2018
IEEE, 2018, pp. 100-105.
[27] M. Gabardi, D. Chiaradia, D. Leonardis, M. Solazzi, and A. Frisoli,
"A High Performance Thermal Control for Simulation of Different
Materials in a Fingertip Haptic Device," in International Conference
on Human Haptic Sensing and Touch Enabled Computer
Applications, 2018, pp. 313-325.
[28] M. Nakatani, K. Sato, K. Sato, Y. Kawana, D. Takai, K.
Minamizawa, et al., "A novel multimodal tactile module that can
provide vibro-thermal feedback," in International AsiaHaptics
conference, 2016, pp. 437-443.
[29] K. Sato and T. Maeno, "Presentation of sudden temperature change
using spatially divided warm and cool stimuli," in International
Conference on Human Haptic Sensing and Touch Enabled
Computer Applications, 2012, pp. 457-468.
[30] S. Gallo, L. Cucu, N. Thevenaz, A. Sengul, and H. Bleuler, "Design
and control of a novel thermo-tactile multimodal display," in
Haptics Symposium (HAPTICS), 2014 IEEE, 2014, pp. 75-81.
[31] H. Huang, T. Li, C. Antfolk, C. Enz, J. Justiz, and V. M. Koch,
"Experiment and investigation of two types of vibrotactile devices,"
in Biomedical Robotics and Biomechatronics (BioRob), 2016 6th
IEEE International Conference on, Singapore, Singapore, 2016, pp.
1266-1271.
[32] K. Bark, J. W. Wheeler, S. Premakumar, and M. R. Cutkosky,
"Comparison of skin stretch and vibrotactile stimulation for
feedback of proprioceptive information," in Symposium on Haptic
Interfaces for Virtual Environments and Teleoperator Systems
2008, Reno, Nevada, USA, 2008, pp. 71-78.
[33] C. Antfolk, M. D'Alonzo, M. Controzzi, G. Lundborg, B. Rosén, F.
Sebelius, et al., "Artificial redirection of sensation from prosthetic
fingers to the phantom hand map on transradial amputees:
vibrotactile versus mechanotactile sensory feedback," IEEE
transactions on neural systems and rehabilitation engineering, vol.
21, pp. 112-120, 2013.
[34] H. Huang, T. Li, C. Bruschini, C. Enz, J. Justiz, C. Antfolk, et al.,
"Multi-modal Sensory Feedback System for Upper Limb
Amputees," in 2017 New Generation of CAS (NGCAS), Genova,
Genoa, 2017, pp. 193-196.
[35] E. Romero and D. Elias, "Design of a non invasive haptic feedback
device for transradial myoelectric upper limb prosthesis," in
ANDESCON, 2016 IEEE, Arequipa, Peru, 2016, pp. 1-4.
[36] A. Ajoudani, S. B. Godfrey, M. Bianchi, M. G. Catalano, G. Grioli,
N. Tsagarakis, et al., "Exploring teleimpedance and tactile feedback
for intuitive control of the pisa/iit softhand," IEEE transactions on
haptics, vol. 7, pp. 203-215, 2014.
[37] M. C. Jimenez and J. A. Fishel, "Evaluation of force, vibration and
thermal tactile feedback in prosthetic limbs," in Haptics Symposium
(HAPTICS), 2014 IEEE, Houston, TX, USA, 2014, pp. 437-441.
[38] P. Kapur, M. Jensen, L. J. Buxbaum, S. A. Jax, and K. J.
Kuchenbecker, "Spatially distributed tactile feedback for kinesthetic
motion guidance," in Haptics Symposium 2010, Waltham,
Massachusetts, 2010, pp. 519-526.
[39] J. Wheeler, K. Bark, J. Savall, and M. Cutkosky, "Investigation of
rotational skin stretch for proprioceptive feedback with application
to myoelectric systems," IEEE Transactions on Neural Systems and
Rehabilitation Engineering, vol. 18, pp. 58-66, 2010.
[40] M. N. Nemah, C. Y. Low, O. H. Aldulaymi, P. Ong, and A. A.
Qasim, "A Review of Non-Invasive Haptic Feedback stimulation
Techniques for Upper Extremity Prostheses," International Journal
of Integrated Engineering, vol. 11, 2019.
[41] H. Ritter, R. Haschke, and J. J. Steil, "A dual interaction perspective
for robot cognition: grasping as a “rosetta stone”," in Perspectives
of neural-symbolic integration, ed: Springer, 2007, pp. 159-178.
[42] D. P. Cotton, P. H. Chappell, A. Cranny, N. M. White, and S. P.
Beeby, "A novel thick-film piezoelectric slip sensor for a prosthetic
hand," IEEE sensors journal, vol. 7, pp. 752-761, 2007.
[43] M. N. Nemah, C. Y. Low, P. Ong, and N. A. C. Zakaria,
"Development and evaluation of a spot sensor glove for the tactile
prosthetic hand," International Journal of Engineering and
Technology (UAE), vol. 7, pp. 63-69, 2018.
[44] D. P. PVT.LTD. RAISE 3D N2 PLUS. Available:
https://www.ithink3dp.com
[45] A. M. Donald and E. J. Kramer, "Plastic deformation mechanisms
in poly (acrylonitrile-butadiene styrene)[ABS]," Journal of
Materials Science, vol. 17, pp. 1765-1772, 1982.
[46] Peratech. (November 2015). QTC SP200 Series Datasheet SP200-
05 Series, Single Point Sensors. Available:
https://www.peratech.com
[47] C. Antfolk, A. Björkman, S.-O. Frank, F. Sebelius, G. Lundborg,
and B. Rosen, "Sensory feedback from a prosthetic hand based on
air-mediated pressure from the hand to the forearm skin," Journal
of rehabilitation medicine, vol. 44, pp. 702-707, 2012.
[48] OnlineShop. (2018). Micro Servo 5-10g / SG90 Analog. Available:
http://www.towerpro.com.tw
[49] PrecisionMicrodrives. (2018). 10mm Linear Resonant Actuator.
Available: https://www.precisionmicrodrives.com
[50] L. C. Y. Mohammed Najeh Nemah, Ong Puluy,, "A vibrotactile
prosthetic device for detection of contact pressure and surface
texture in upper extremity," In progress: International Journal of
Advanced Robotic Systems, 2019.
[51] J. M. Fontana, R. O’Brien, E. Laciar, L. S. Maglione, and L.
Molisani, "Vibrotactile Stimulation in the Upper-Arm for Restoring
Individual Finger Sensations in Hand Prosthesis," Journal of
Medical and Biological Engineering, pp. 1-8, 2018.
http://www.ithink3dp.com/http://www.peratech.com/http://www.towerpro.com.tw/http://www.precisionmicrodrives.com/