leap motion for sterile manipulation of 3d models
TRANSCRIPT
Leapmotionforsterilemanipulationof3DmodelsEuropeanProjectSemesterGroupF
AutumnSemester2017
Supervisor: LouiseOram
Team: GeorgBaumann
TimFink
HubertMarret
JosepMariaPujolMarch
2
3
Abstract
Performingsurgerieswith3Dmodelsaretoanincreasingextentbecomingnormalpractice.
Thecommontoolformanipulatingthese3Dobjectsiswithacomputermouse,whichisnot
designed for a three-dimensional virtual space. Thus, the use of a computer mouse in
combinationwith3Dobjectsisnotintuitive.Thisisthereasonforexperimentingwithnew
inputdevices,suchastheLeapMotiondevice, toenhancetheworkflowofcontrolling3D
objects.Therefore,theprojectdevelopedaprogramformanipulatingthesemodelsinUnity
with the Leap Motion device. A user survey was conducted to identify issues and help
understandhowthesurgeonsuseandinteractdirectlywitha3Dobjectinasterilemanner.
The results of this project strongly support the fact that theusageof the LeapMotion is
moreefficient in various tasks thana computermouse.With this the LeapMotion stacks
uponrevolutionary2Dimaginginmedicalpractice,andthusisanimportantpieceinfuture
medicalcareandsurgery.
4
Acknowledgments
Theprogresswemadewiththeproject“Leapmotionforsterilemanipulationof3Dmodels”
required a great amount of guidance and assistance by students, professors ofHiOA and
staffmembersoftheRikshospitaletUniversityHospital.Wefeltprivilegedtodoresearchon
project,whichisvitalforsurgery.
First,ourdeepestappreciationgoestooursupervisorLouiseOramwhosesupport,guidance
andsolicitudewereinnumerablyvaluablethroughoutthesemester.
Second,wewouldliketothankourprofessorsNinaHagerup,TengelAasSandtrøandPetter
Øyan,fortheirwellorganizedandsupportinglectures.WearealsogratefulofBritBalgaard,
forhercontinuoussupportthroughoutthesemester.
Our gratitude also needs to be expressed to the staff members of the Oslo University
Hospital. Louise Oram established communication between the hospital and the project
team.Inaddition,wefeltprivilegedtoobservetwosurgeries.
WealsothanktheinformationtechnologyconsultancySopraSteriaforlettingusvisitoneof
themostbeautifulworkingplacesinOslo.Besides,theyprovidedusconstructivefeedback
andthepossibilitytotestmixedrealitydevices,suchastheHololens.
Ourprojectteamisverygrateful,thattheHøgskoleniOsloogAkershusmadethisproject
possibleforforeignstudents,byofferingaEuropeanProjectSemester.
Wecannotforgettothankallparticipantswhoparticipatedinourusersurvey.
5
Glossary
SDK ASoftwareDevelopmentKitisasetofsoftwaretools.
3Dobject Amathematicalrepresentationofanysurfaceofanobjectinthree-
dimensions.
collider Aninvisibleframeofa3Dobjectwhichdetectscollisions.
Unity Agameenginefordevelopingvideogamesandsimulationsfor
multipleplatforms.
C# Aobject-orientedprogramminglanguage.
LeapMotion AnAmericancompanywhichmanufacturesasensordevicefor
handmovements.
HiOA OsloandAkershusUniversityCollegeofAppliedSciences
Unix Afamilyofvariousoperatingsystems.
6
7
TableofContents
1. Introduction ___________________________________________________________9
2. ProjectEvaluation______________________________________________________10
2.1. Relevance ______________________________________________________________10
2.2. Effectiveness____________________________________________________________10
2.3. Impact_________________________________________________________________11
2.4. Stakeholderanalysis______________________________________________________12
3. Methods_____________________________________________________________13
3.1. Assessmentoftheproject’sinitialsituation____________________________________13
3.2. Development____________________________________________________________14
3.2.1. Features ___________________________________________________________14
3.2.1.1. Rotation ___________________________________________________________14
3.2.1.2. Scaling_____________________________________________________________14
3.2.1.3. GrabandMove______________________________________________________15
3.2.1.4. ResetButton________________________________________________________15
3.2.1.5. Featurecohesion ____________________________________________________17
3.2.2. Programming _______________________________________________________17
3.2.2.1. WorkingwithUnity3D_____________________________________________________173.2.2.2. GitHubandSourceTree____________________________________________________173.2.2.3. Workflow_______________________________________________________________183.3. Usersurvey _______________________________________________________________18
4. Results_______________________________________________________________34
4.1. Programfunctionality_____________________________________________________34
4.1.1. Rotation______________________________________________________________34
4.1.2. Grabandmove________________________________________________________34
4.1.3. Zoom________________________________________________________________35
4.2. Structureofthecode _____________________________________________________35
4.2.1. Classes_____________________________________________________________36
4.3. Resultsofthesurvey______________________________________________________36
4.3.1. GeneralInformation__________________________________________________37
4.3.2. Results_____________________________________________________________40
8
4.3.2.1. Translation _________________________________________________________41
4.3.2.2. Rotation ___________________________________________________________43
4.3.2.3. Zooming ___________________________________________________________45
4.3.2.4. Button_____________________________________________________________47
4.3.2.5. Additionalstatements ________________________________________________48
5. Discussions___________________________________________________________51
5.1. SurveyAnalysis __________________________________________________________51
5.2. FeatureAnalysis _________________________________________________________52
5.2.1. Grabandmove________________________________________________________52
5.2.2. Rotation______________________________________________________________52
5.2.3. Zoom________________________________________________________________53
5.2.4. Button_______________________________________________________________53
5.3. Suggestions_____________________________________________________________53
5.4. Generalanalysis _________________________________________________________54
6. Conclusion____________________________________________________________56
7. PersonalReflection(Belbin)______________________________________________57
8. References ___________________________________________________________60
9. Illustrationdirectory____________________________________________________61
AppendixA:SourceCode_________________________________________________________63
Loader.cs ___________________________________________________________________63
ButtonManager.cs____________________________________________________________64
ButtonVisibilityManager.cs_____________________________________________________68
Rotation.cs__________________________________________________________________69
Translation.cs________________________________________________________________71
Zoom.cs ____________________________________________________________________74
UIManager.cs________________________________________________________________75
Gesture.cs___________________________________________________________________77
ResetButton.cs_______________________________________________________________78
Singleton.cs _________________________________________________________________79
AppendixC:InstallationManual ___________________________________________________81
AppendixC:MethodsUserStudy __________________________________________________82
ScriptforpresentingtheLeapMotionfeatures _____________________________________82
AppendixD:ResultsUserStudy____________________________________________________83
9
1. Introduction
Leap Motion is a computer hardware device which can sense hand movements. Its
applicationisversatile,yetstillcommonlyusedasaninputdeviceforcomputergames.The
UniversityHospitalofOslorecognizedthepotentialoftheLeapMotionDevicetoimprove
theirworkflowin2013-14andstartedresearchintoit.Theyproposedanideaforaproject
totheOsloandAkershusUniversityCollegeofAppliedSciences.Thus,studentsareableto
strengthentheirskillsinaprofessionalenvironmentandhelpsucceedingtheproject.
The mission of the project “Leap motion for sterile manipulation of 3D models” was to
create a program combinedwith a LeapMotion device, inwhich one canmanipulate 3D
modelsoforganswithouttouchingacomputermouse.
TheLeapMotionDeviceandthegameengineUnity3Dwereusedtodevelopaprogramfor
manipulating3DmodelsinUnity.Theprogramshouldallowtheusertointeractwitha3D
model in various ways. A major goal of the project was to provide cross-platform
compatibilityonmajoroperating systems (Linux,MacOS,andWindows).Themainaimof
thegroupwastodevelopafunctionalprototype,whichonecanfurtherdevelopafterthe
project’stermination.Toensurefurtherdevelopmentoftheproject’sprogram,theproject
groupthoroughlydocumentedthesoftwarecode,includingthethoughtandmotivationof
varioustakenactions.
Additionally,ausersurveywiththefinalprototypewasconducted.Clarificationoftheuser
survey’smethodologyandthediscussionofitsresultsareavailableinthereport.
The report contains a project evaluation, the methodology of the user survey and the
developed program aswell as the results. At the end of this report there is a discussion
about thedevelopedprogramand its implemented features and gestures.Moreover, the
reportincludestheproject’ssustainabilityafterthetermination.
10
2. ProjectEvaluation
2.1. Relevance
Assessingthepresentsituationinwhichsurgeonsperformsurgeryisvitaltoidentifyissues
anddesiresforthisproject.Uponthisassessment,theprojectteamwasabletoformulate
achievablegoalsandspecificobjectives.
Presently surgeons must remain sterile while performing surgery, thus various tools and
objects that they get in contact with have to be sterile too. In that case, operating a
computermouseandakeyboardinordertomanipulate3Dobjectsisdonebyanassistant
oranurse.Basedonapreviousexperiment(Balakrishnan,Retal.,1997),executingtasks,
suchasrotatingormovinga3Dobjectofanorganwithacomputermouseisnotintuitive
andergonomic.Thisbecomesfurtherapparentwhenthesurgeondirectshisorherassistant
tomanipulate the 3D object. Describing a certainmove of a 3D images instead of a 2D
imageappearstobemorecomplex.Hence, it ismore likely tomisunderstandeachother,
resultinginpossibleerrors.
In addition,onemight argue that the current situationofmanipulating3Dobjectswitha
computermouserequirespatienceandtime.Becauseofthat,itiscleartosaythatthefocus
shifts from the patient tomanipulating a 3D object. The efficiency of performing various
tasks in 3D space can be enhanced by utilizing the computermousewith a keyboard, to
executeshortcuts,thusimprovingtheoverallworkflow.Itseemsunlikelythatthisapproach
is used by the majority of surgery assistants or nurses, due to the presumably lack of
knowledge in thesoftwareusedfordisplaying3Dmodels. Inaddition, thegroupbelieves,
the lacking motivation of increasingly utilizing 3D objects prevents it of enhancing the
existing workflow. It is obvious to say, that medical imaging for the human body
revolutionized clinical practice in the past ten to twenty years. This project aims to build
uponpresenttechniques.
2.2. Effectiveness
Evaluating the effectiveness of various approaches and outcomes in correlation to the
projectdesigniskeytounderstandthejustificationofcertainactions.
11
Inthefirstquarteroftheproject'stimeframetheprojectmembersagreeduponimproving
thepriorprojectinUnity.Limitedresourcesanddocumentationrestrictedtheapproachto
enhance the software. Since the software was incompatible with the project’s members
Unix computers, it is safe to say that this approachwould not havemet the initial aims.
Thus, theproject teamdecided to reprogram the software inUnitywithanolder SDK, in
ordertoprovidecross-platformcompatibility.
At this moment, the latest SDK was exclusively on Windows available. Consequently,
compromises had been made by selecting an older software development kit. Various
featuresandfunctionalitywerenotavailableintheSDKversiontheprojectteamchose.This
approach facilitated developing the program and dividing tasks between all four group
members.
This group received supportive feedback from two mentor meetings with professors of
HiOA,inordertoassessandsolveissuesoftheproject.Inaddition,allfourgroupmembers
attended courses in English and Project Management throughout the project cycle. The
project’ssupervisor,LouiseOram,arrangedameetingwiththeproject’scoremembersand
staff members of the Oslo University Hospital Rikshospitalet, in order to see a surgery.
Understanding the environment in which a surgery is being performed was crucial for
designingthesoftware.
2.3. Impact
To ensure the project’s long-term impacts, various objectives have been executed
successfully.Firstly,theprogramwaswrittenintheobject-orientedprogramminglanguage
C#, appropriate for Unity. Secondly, the source code of the software is thoroughly
documented.Lastly,instructionsforinstallingthesoftwarehavebeenmadepubliconlineto
ensureafullyfunctionalsoftwareonmultipleoperatingsystems.
Itiscertaintosaythattheproject’soutcomeswillbenefitsurgeonsandnursesdirectly,due
totheimprovedworkflow.Patientsareconsideredtobeindirectbeneficiaries.
12
2.4. Stakeholderanalysis
The following paragraphs will present the main stakeholders, from core and primary to
secondary.
OuruniqueCore,LouiseOram,worksinthehospitalRikshospitaletasacomputerscientist.
Intheproject,sheactsasanexternalsupervisor.SheproposedtheLeapMotionprojectand
assisted the group by giving continuous constructive feedback and establishing
communicationbetweenRikshospitalet, thecompanySopraSteria,acompanyspecializing
invirtualreality,andtheprojectgroup.
Primary Stakeholder: Rikshospitalet is considered as the project’smain client, due to the
developmentofasoftware,designedforsurgeonsworkingthere.
SecondaryStakeholders:HiOA,theUniversityCollegeprovidesthepossibilitytoenrollinan
European Project Semester. Patients which will be treated by surgeons who use the
project’ssoftwaretoviewthe3Dmodels,arealsoconsideredsecondaryStakeholders.The
EuropeanCommission,whichofferstheEPS,mightalsobeinterestedintheresultsofthis
project. Potential customers, suchasotherhospitals thatwant theproject’s product, can
alsobeconsideredpartofthisgroup.REK,theRegionalCommitteesforMedicalandHealth
Research Ethics, is also considered a secondary stakeholder. Finally, the company Sopra
Steria,whichsupportedtheprojectwithcreativeapproachesandfeedback,isanimportant
stakeholder.
Atthebeginning,theprojectteamdecidedtoconsiderprofessorsasstakeholders,butthey
wereratherconsideredasadvisors,guidingtheworkandgivingfeedback.
Theprimarystakeholdersaremainlyinterestedinutilizingtheprograminthenearfuture.
Theiraimistousetheprogramtoenhancetheworkflowofsurgeries,thussavingvaluable
timeinsurgeriesandmedicalpractice.
The secondary stakeholders are rather keen to see the project’s success. They are not
necessarily concerned about how the program functions, rather the utility of it and the
impacttheprogramhasonitsenvironmentismoreimportantforthem.
13
Theprimarystakeholdersareactiveparticipants,whichcaninfluencetheprojectsdecisions.
Staffmembersof theHiOACollegeonly informtheproject teamingivingthemfeedback,
henceshapingtheoutcomeoftheproject.
3. Methods
3.1. Assessmentoftheproject’sinitialsituation
Thecurrentprojectgroupworkingontheproject“Leapmotionforsterilemanipulationof
3D models” is not the first group working on this project. There have been two groups
before,who participated in the European Project Semester, working on this project. The
members of this team received the results of the groups, which were working on this
project in the previous semesters. The initial idea was to continue working on the
prototypes and results given by the previous project groups. The previous groups were
mainlyworkingonWindows.Sincethedominatingoperatingsysteminthecurrentproject
groupwasMacOS,thecurrentprojectgroupwasnotabletomakethegivenprogramwork
on their computers. To runLeapMotion thecode requiresaSDK (SoftwareDevelopment
Kit)which transforms the signals from LeapMotion into readabledata for the computer.
ThereareseveralSDKs,whichhavebeendevelopedbytheLeapMotioncompanyoverthe
pastyears.Allofthemhavethesamefunctionality,butthenewertheversionisthemore
optimizationsandnewfeatureshavebeenimplementedtotheSDK.ThelatestSDKatthis
time,whichrunsonWindowsandMacOSistheSDK2.3.ThenewestSDK,calledOrion,only
runsonWindows.ThreeofthefourmembersoftheteamownanApplecomputer.Thefirst
ideawas touse theSDKOrion since itwas thenewest SDK,but thiswould require three
teammemberstoinstallWindowsontheircomputersorrunitasavirtualmachine(VM).It
canbemuchslowertorunprogramsinsideaVM,whichwouldhavecausedtheproblem,
thatUnitymightnotworkasfluentasitshould.Afterthat,theteamagreedtousetheSDK
2.3evenifitwasolderthantheSDKOrion.
14
3.2. Development
3.2.1. Features
As thegoalof theprojectwas tobuilda simpleprototype, the teamchose to implement
only the basic functions for 3D model manipulation: Rotating, Translating and Scaling.
However, the Translationmovement has evolved to bemore like aGrab andMove for a
betteruserexperience.
Moreover, another feature had to be implemented to help the user interact with the
application:abuttontoresetthemodeltoitsinitialstate.
3.2.1.1. Rotation
Objectiveofthefeature
The 3D model is static. To see the parts of the organ that are obscured in the current
viewingangle,arotationoftheobjectwillbeneeded.
Conception
Thefirstideawastorotatetheobjectbymovingthehandaroundit,butnottouchingthe
object.The3Dmodelwouldfollowthehand’srotationasitisgrabbedfromadistance.The
main problem of this interaction method is that it would be unclear for the user if the
gestureisactivated.
After reflection, a second idea was to put the hand inside the object and rotate it. The
modelwillmimicthehand’srotation.
Thesecond ideawaseasier to implement,andappearedmore intuitive for theteamso it
wasthegesturechosenforthisfeature.
3.2.1.2. Scaling
Objectiveofthefeature
The3Dorgancanbereallycomplex.Thus,surgeonsmaywanttozoomonaspecificpartof
themodelforabettervisibility.
15
Conception
Atfirst,theideawasapinchgestureinsidethemodelwithonehand.Thentheusersimply
hastomovethehandpinchingtowardorawayfromtheuser.
Asecondideawasapinchgesturewithbothhandsinsidethemodel.Thenthemovement
wouldbetoseparatethehandspinchingtozoomin,orbringthemclosertozoomout.This
actionwouldscalethe3Dmodel:enlargingittozoomin,shrinkingittozoomout.
Thisconceptprovidesmoreadvantagesthanthefirstone.Themovementwouldmimicthe
zoom gesture on a tablet or smartphone, an action nowadays omnipresent. Besides, the
concept would avoid displacing the viewpoint. Thus, the viewpoint would keep its static
behavior.
3.2.1.3. GrabandMove
Objectiveofthefeature
At thebeginning,no translationof theobjectwas required.However, ifa zoomhasbeen
performed,thesurgeonsmaywanttolookatanotherpartoftheorganthatisnotvisible.
Theavoidzoomingout,atranslationmightbeexecuted.
Conception
Atfirst,theideawasfistgestureinsidethemodel.Thentheuserwouldonlymovethefistto
dragtheobject.Thedragwouldonlytranslatethemodelwithoutanyrotation.
Nonetheless,thefistgesturewasnotwelldetectedbytheLeapMotion.Athoughtwasto
adopt the pinch gesture used for zooming. The pinch would act as a grab gesture. The
movementthenwouldexecutethesametranslation,butwouldalsobeassociatedwiththe
rotation.Thisfinalgesturewouldrepresentareal-lifebehavior:grabbinganobjectwiththe
handandmovingitaround.
3.2.1.4. ResetButton
Objectiveofthefeature
16
The3Dobjectwillbedisplacedinthe3Dspaceandthecameraisstatic.Thus,afterseveral
manipulations,theorganmightbedisplacedinthecornersofthescreenorevenoutofit.
The Leap Motion controller does not have a wide range of detection. Therefore it is
complicatedtohavethehandspreciselydetectedwhentheyareatthescreen’sedges.To
avoidthisproblem,onesolutionwastocreateabuttontoresettheobjecttoitsinitialstate.
Itsposition,rotationandscalewouldbereset.
Conception
Toactivateabuttonwithvirtualhands ina3Dapplication isnotacommontask. Indeed,
people nowadays aremostly utilizing their hands to interactwith 2D user interfaces, for
examplesmartphonesandtablets.Therefore,severalapproacheswerestudied.
The Leap Motion Playground applications were tested to study the different ways of
handlingabuttonwithvirtualhands.Inthesecases,thebuttonswere2Dicons.Toactivate
one, the user had to hover his index finger one second on top of it. This solution was
sometimesworkingperfectly.However, itwasoccasionallydifficult toactivate thebutton
withthefingerhoveringoveritwhilenothingwashappening.Hence,anotherapproachhad
tobefound.Thethoughtwastocreatea3Dbuttonandtousethephysicsofthehandto
push it.AnarticlefromFelixNoller(2016),ahumanfactorresearcher,explainedawayto
implementa3DbuttonusingLeapMotionandUnity.Thebuttonispushedbythehandand
is goingback to its original positionwhen thehand leaves it. This last solutionwasmore
ergonomicasonly3Dinteractionwillavailablefortheusers.Theywillnothavetodealwith
both2Dand3Dinteractions.
Figure1:handpushinga3Dbutton
17
3.2.1.5. Featurecohesion
Thefeatureshavebeenconceivedtobeeasilyassociatedwithothers.Therotationcanwork
alone,butisexecutedaswellwhentheusergrabsthemodel.Besides,thezoomfeatureis
usingthepinchgestureofthegrabandmove.
Moreimportantly,thefeatureshavebeenimplementedtobeusedinasamemanipulation,
oneafteranother.Thegoalhereistogivetheuserasensationoffreedommanipulatingthe
organ:as itwasa realobjecthecouldgrab,bringcloser tohiseye, rotateandscalewith
intuitivegestures.Theresetbuttonwouldbepresentheretograntarapidwaytogoback
toastandardpositionofthe3Dobjectandhelptheuseravoidfeelinglost.
3.2.2. Programming
3.2.2.1. WorkingwithUnity3D
Thepreviousgroup thatworkedon theproject chose toassociateUnity3Dwith the Leap
MotionSDK.Unity3D(mostcommonlycalled“Unity”)isapowerful3DEngineusedmostly
for game development but also for any 3D physical interaction software. It has an
ergonomiceditorandacomplexAPI(setofclasses,methodsandfunctionsthatcanbeused
forprogramming).Amultitudeofassetsandpluginsmadebydevelopersarealsoavailable
toaddmorefeaturestotheengine.OneofthemistheLeapMotionplugin,itretrievesthe
LeapMotiondataandgivesittoUnity. Italsoallowsthedevelopertocreatethehandsin
the3Dscenesoitcanplaywiththe3Dmodels.
With all these advantages,Unity has been kept by the team and used for the software
development.
3.2.2.2. GitHubandSourceTree
Whenasoftwareisdeveloped,it isessentialtoutilizeaversioncontroltool.Thiswillsave
severalversionsof thesoftwareonaserver, theymightbeseenasbackups.Therefore, if
oneorvariousfilesarehavingproblems,theycanberestoredtoapreviousversion.
18
Besidesthis,versioncontrolisanefficienttoolforteamworking.Everydeveloperisableto
developonthesameprojectwithouthavingcodeconflictswiththeothers.
Tochoosewhichtoolshouldbeusedfortheapplication,theexperienceddevelopersofthe
team selected Git with the GitHub platform. A web platform used to store and share
projectswithothercoders.Aprojectmanagementtoolispresentinthewebsiteaswell.In
addition, the software SourceTree has been chosen to provide aGraphical Interfacewith
Git.
3.2.2.3. Workflow
Theprojecthasbeenrealized in3phases:Analysis,ProgrammingandSurveying.Foreach
part,thegroupwassplitintosubgroupsorindividualsworkingonsubcomponentstogainin
efficiency.
Firstly,theteamhadtoanalyzetheprojecttofinditsgoals.Theapplicationoftheprevious
grouphasbeenanalyzedtoseewhatcanbedonewithit.Severalresearchesontheinternet
havebeendoneonLeapMotiontoexpandtheteam’sunderstandingonthesubjectandto
look for various approaches. Finally, one teammember learned how to work with Unity
followingafewtutorials.
Secondly,when thegoalsweredefined, the teamstartedcodingon theapplication.Each
memberhadafeaturetoimplement.
Lastly,when a first prototypewas running, a part of the team startedworking on a user
survey.ThegoalwastotesttheapplicationandtostudytheefficiencyofLeapMotion.The
other members continued coding on the application for optimization and to finalize the
documentation.
3.3. Usersurvey
3.3.1. Methodsoftheusersurvey
Aftermostoftheprogramminganddevelopmentpartwasdonetheprojectgrouphadthe
ideatoconductausersurveytoassesspeople’sfirst-timeexperiencewiththeLeapMotion
device.Thereareseveralmotivationstocarryoutausersurvey:
19
1. Thegoalwastosee,iftheimplementedfeaturesareintuitiveorifsomegesturesor
featuresneed tobe reconsidered. It is alwaysnecessary todoearly-stage testingofa
createdproduct,soonecanidentifymistakes,flawsanddesignissues.Timeandmoney
waslimited,sotheprojectgroupwantstogiveaconstructivefeedbacktotheupcoming
groupworkingonthisprojectaboutthecreatedfeatures.Thethoughtbehindthisisto
establishacomfortableprojectstartfortheupcominggroup.
2. By lookingathowpeople interactwithaprototypeofone’sproductonewillgeta
greatamountof information,which isbeneficial forfuturechanges.Conductingauser
survey andUser testing reduces time andmoney investment. By having users testing
one’sproductyouwillgetfeedback.Inthecaseoftheprojectgroup,thefeedbackwas
notonlygivenbyspeakingtothepeople,whoparticipatedintheusersurvey,butalso
byobservinghowtheparticipantsutilizedtheLeapMotiondeviceincombinationwith
the developed program. The information that was collected is important for the
upcominggroupworkingonthisproject.
3. Byconductingausersurvey,onegetsanunbiasedperspectiveofotherpeople.Due
to the biased perspective of the project team members, it was necessary to gather
feedbackandopinions frompeopleoutsidetheproject team.Thereasonfor this is to
spotpossible flawsoftheproduct,whichmightnothavebeenobviousfortheproject
team,duetotheirbiasedperspective.
SigningaConsentFormtoparticipateintheusersurveywasrequired.
TheConsentFormincludedthefollowinginformationandcanbefoundintheappendix.
·Backgroundandpurposeofthesurvey.
·Whatdoesparticipationinthesurveyimply?
·Whatwillhappentotheparticipant’sinformation?
·Aconfirmation,thatitisvoluntarytoparticipateinthesurvey.
In the followingpart theuser surveywill beexplained. Theuser surveywas createdwith
GoogleForms.Inaddition,screenshotsoftheusersurveywillbedisplayed.
20
3.3.2. GeneralInformation
Figure2:GeneralInformationabouttheLeapMotionProject
In the beginning of the user survey the project group briefed each participant about the
LeapMotionprojectandthegoalsofthesurvey.Itneedstobementioned,thatsomeofthe
followingquestions requireananswer,whileothersdonot.Questions,which requiresan
answeraremarkedwitharedstar.At thestartof thesurvey itwasmentioned, thatonly
honestopinionsandanswerstothequestioncanhelptheprojectimprovingtheprogram.
3.3.3. PersonalInformation
21
Figure3:Startofthecollectionofpersonalinformation.Askingaboutthegender.
In this part the participants should click whether they are male or female. It was also
allowedtorefusegivinganinformationaboutyourgenderbyclicking“Abstain”.
Thegoalofthisquestionwastoseeifacertaingenderissignificantlybetterinworkingwith
LeapMotion.
Figure4:Askingabouttheage.
Inthispartofthesurveytheparticipantswereaskedabouttheirage.Thereasonforasking
for age brackets, and not for the specific age of the survey participant,was tomake the
surveymoreanonymous.
Theseagebracketswerechosenforaspecificreason.Thebracket18-25 isconsideredfor
students,whomightbe in theirbachelor’sprogram.Thebracket26-40 ismeant forolder
studentsandpeople,whomighthaveajob.Theoption“Olderthan40”ismeantforpeople,
who did not grow up with current modern technology and therefore may be less
comfortableorfamiliarwithit.Thereasonforthisquestionistosee,whichagebracketcan
handletheLeapMotionprogramthebest.
22
Figure5:Askingabouthowoftenthesurveyparticipantisplayingvideogames.
Heretheparticipantisaskedabouthowfrequentlyoneplaysvideogames.Themotivation
forasking thisquestionwas to see ifpeople,whooftenplayvideogamesare superior in
interactingwiththeLeapMotiondeviceincombinationwiththeproject’sprogram.
Figure6:Askingaboutwhichvideogamestheparticipantplays.
Thereisasimplereasonforaskingthisquestion.Theparticipantmightsaythatsheorheis
playingvideogameseveryday.Ifoneisalreadyexperiencedinplaying3Dgames,heorshe
mightperformbetter,duetoabettersenseoforientationinvirtual3Dspace.Thegoalof
the questions in Fig x+3 and Fig x+4was to find out if oneplays video games andwhich
video game, to see if it would help them interactingwith LeapMotionmore swiftly and
reliable, so the project group could see if they are acting more intuitive with the Leap
Motiondevice.
23
Figure7:Askinginhowmanydimensionsthevideogamesareplayed.
Thisquestionisasimplifiedversionofthepriorquestion.Ifparticipantsareuncertainabout
fillingout thepreviousquestionwitha specificgame title, thisquestionbroadly indicates
whetheroneisfamiliarwitha3Dora2Denvironment.
Figure8:AskingabouthavingexperiencewithLeapMotionorVirtualReality.
There isasimplereasonforaskingabouttheparticipanthavinganyexperiencewithLeap
MotionorVirtualReality.Theprojectgroupexpectedthose,whohavemadeanexperience
withthesethingstointeractbetterandmoreexperiencedwiththeLeapMotiondevice.
Figure9:Askingiftheparticipanthasexperienceinmanipulating3Dmodels.
Thethoughtbehindaskingthisquestionwastoseeifparticipants,whoareexperiencedin
interactingwitha3Dmodelontheircomputerwiththeircomputermouse,wouldalsobe
betterinmanipulatinga3DmodelwiththeLeapMotionDevice.
3.3.4. LeapMotionDemonstrationandexplanationofthevirtualenvironment
Beforeexplaininghowtheusersurveywillbecontinued,thevirtualenvironment,inwhich
theparticipantofthesurveywillwork,needstobeintroducedandexplained.
24
Figure10:Virtualenvironment,inwhichtheparticipantofthesurveyinteracts.
Thisisthevirtualenvironmenteveryparticipantofthesurveywasworkingin.Thefollowing
partwillexplain,whatthenumbersinthepicturearereferringto.
1. Thisisa3Dmodelofadice.Theparticipantofthesurveywillmostlyworkwiththe
diceduringtheusersurvey.Twofeatureswerelinkedtothedice.Itcanbetranslated,
rotatedandscaledbyusingcertaingestures.
2. AHandappearsinthevirtualenvironment,iftheparticipantofthesurveyismoving
hisorherhandabovetheLeapMotiondevice.
3. Abuttonwillappear,iftheparticipantismovinghisorherhandintotherightpartof
the virtual environment. If one’s hand is elsewhere, the button will disappear. The
buttonresetstheposition,therotationandthescaleofthediceoncepushed.
4. “Hva heter du?” is written in a small font under the number six of the dice. This
phrasewillbepartofanupcomingtasktheparticipantsofthesurveyareaskedtosolve.
Thevirtual environmentas it canbe seen in Fig x+9was thedefault virtual environment,
displayedeverytimetheprogramwasstarted.
25
Figure11:IntroductionbeforetheLeapMotionprogramwasdemonstrated.
Each participant followed a procedure. Features, such as translation, rotation, scaling
(zooming) and resetting the dice, were presented. A script of how to demonstrate the
featurescanbefoundintheappendix.
It was important, that no phrases were used, that might push the developed program
towardsapositiveoutcome.
3.3.5. Usertesting
Figure12:Introducingtheupcomingtasks.
InordertoanalyzehowwelltheLeapMotionprogramisworkingfourtaskswerecreated:
·Atranslationtask
·Arotationtask
·Azoomingtask
·Abuttontask
Thegoalwastotesteveryfeature,theprojectgroupdeveloped,indifferentshorttasks.It
needs tobementioned, that the taskswerecounterbalanced.The reason for that is, that
theparticipantsaregainingexperienceabouthowtouseLeapMotionprobablyduringthe
tasks.Ifyouwouldhavethesamelasttaskforeveryparticipant,thetimeparticipantswould
26
needtosolvethistaskwouldalwaysbelowerthanthetimetheywouldneed,iftheywere
doing this task for the first time. TheButtonTaskwas always askedafter the Translation
TaskbecauseitwasrequiredtoknowhowtotranslateforsolvingtheButtonTask.
Thefollowingsequenceswereused:
Tabley:Sequencesusedforthetasks
Rotation Translation Zoom
Translation Zoom Rotation
Zoom Rotation Translation
Foreverytask,thetimewastakenandfilledintoafieldbeneaththedescriptionofthetask.
3.3.6. Translation
Figure13:ExplanationoftheTranslationTask
Thegoalofthetranslationtaskwastopinchwiththefingersandbeinsidethedice,sothe
participantcouldgrabit.Aftergrabbingthedice,thediceshould“touch”thecornersofthe
screeninthesequence,thatwasmentionedinFigx+11.
27
Figure14:movementprocedure
3.3.7. Rotation
Figure15:Explanationoftherotationtask
28
Thistaskaimedtoassesstheunderstandingoftheparticipantbetweenpinchinginthedice
andnotpinching.Bynotpinchinginthediceandjusthavingyourhandinthemodel,the3D
modelofthedicewouldonlyrotatewithyourhandandnotmove.
Figure16:Rotationofthedice
3.3.8. Zooming
Figure17:ExplanationoftheZoomingTask
29
TheZoomingTaskrequiredparticipants tousebothof theirhands inorder tochangethe
scalingofthedice.Thegoalwastoreadthelittlewrittensentence“Hvaheterdu?”.
Figure18:Diceafterchangingitsscaling.
3.3.9. Button
Figure19:ExplanationoftheButtonTask
Forthistaskthepre-knowledgeoftranslatinganobjectina3Dvirtualspacewasrequired.
Thegoalofthistaskwastosee,iftheparticipantscouldunderstandhowdeepthebuttonis
in thevirtual environmentandhowswiftly they couldmoveaway thediceandpress the
buttontwice.
30
Figure20:Pressingthebutton
3.3.10. Feedback
Figure21:Explanationofthereviewingsystem
In the finalpartof thesurvey, theparticipantwereaskedabout theirhonestopinions.As
youcansee inFigx+19, theparticipantswereable toansweronascale from1to5,one
displaying a strong disagreement, while five represents a strong agreement toward the
question.
31
Figure22:FirsttwostatementsabouttheTranslationfeature.
Therewerealwaystwostatementsgiventoonefeature intheLeapMotionprogram.The
firststatementwasaskedwithanegativeadjective.Thesecondstatementwasaskedwitha
positiveadjective. If theparticipantwassatisfiedaboutafeatureandthought itwouldbe
easytouse,sheorhehadtogivea1ora2inthefirststatementanda4ora5instatement
two.Thethoughtbehindthissystemwastonotlettheparticipantfallintoarhythm,where
sheorhewouldalwayssticktothesamenumbers.
Figure23:StatementsabouttheRotationfeature
32
Figure24:StatementsabouttheZoomingfeature
Figure25:StatementsabouttheButtonfeature
The statements about the Button feature differ compared to the prior statements. The
thoughtisstillthesame.Onestatementwithapositiveadjectiveandonestatement,which
isformulatednegative.
ThefollowingquestionsareaboutLeapMotionandhowtheparticipantsfeltusingit.
33
Figure26:StatementsaboutLeapMotioningeneral
Figure27:AdditionalStatementsandendofthesurveybypressingthesubmitbutton.
34
4. Results
4.1. Programfunctionality
Thefinalproductcontainsvariousgesturesandfunctionalitywhichmightdifferfrominitial
objectives.
4.1.1. Rotation
Torotateanobjectinanydirectiononlyonehandisrequired.Thefunctionwillbeexecuted
upon entering the object with one hand. The collider of the object detects if a hand is
touching or entering it and consequently sets a Boolean variable to true. After that, the
rotationofthehandisassignedtotheobject’srotation.Itistobeobservedthatnogesture
isnecessary toactivate the rotation function.Thus,a reliableandcomfortablemethodof
rotatinga3Dobjectisgiven.
4.1.2. Grabandmove
Inorder tomoveanobject,onehas toperformapinchgesture.Nopre-installedgesture
suited thedemandsof a responsive and reliableway to activate the translation function.
Subsequently, the project’s members developed their individual gesture for pinching.
Wheneverthethumbtipisclosertotheotherfingertipsthanaspecificdistance,aBoolean
variableissettotrue.Onlypinchinginsidetheobjectwillexecutethetranslationfunction.
Whilemovingtheobject,onecanstopthisfunctionbyreleasingthepinchgesture.Grabbing
andmovinga3Dobjectwiththistechniquedoesfeelcomfortableandintuitive,duetoits
natural resemblance in human society. Minor issues with the recognition of the pinch
gesturestilloccur,becauseof thephysical limitations theLeapMotioncontroller inherits.
The device has two wide angle infrared cameras, which send the raw data to the
microcontrollerforfurtherprocessing.Ifthepinchgestureiscoveredbythebacksideofthe
palm,thedevicehasnodirect lineofsightofthefingers,thustheLeapMotioncontroller
couldmisinterpretcertainmovements.Forthemostaccurateandreliableexecutionofthe
pinchgesture,thepalmofthehandshouldfacetheLeapMotiondevice.
35
4.1.3. Zoom
Enlarging an object was certainly the most complex function to implement. At first, the
projectgroupimplementedapinchtozoomgesture,whichisalsopresentontouchscreen
devices.Intheory,theobjectshouldenlargeuponenteringitwithonehandandexpanding
thedistancebetweenthethumbandtheindexfinger.Scalingitdown,onehastominimize
thedistancebetweenthesetwofingers.Ifscaleddown,thehandillustratesapinch,which
wouldinterferewiththegrabandmovefunction.Thus,analternativemethodforscalingan
object was implemented. Presently, two hands are required to execute this function.
Pinchingbothhands insidetheobjectandexpandingthespaceofthehandswillscalethe
object in correlation to the distance of the hands. Therefore, amore reliablemethod of
scalinganobjectisgiven.
4.2. Structureofthecode
AmajorpartismanagedbyUnityandtheLeapMotionAPI.Therefore,thestructureofthe
codeisstraightforwardandadaptable.
Figure28:ObjectDiagramoftheApplication
36
4.2.1. Classes
Loader:AttachedtoaGameObject, it is thescript that loads theUIManagerandthus the
Singleton.
Singleton: It is a design pattern. A class that can be instantiated only once and can be
accessedfromeveryotherscriptinsidetheprogram.ItcontainstheUIManager,theorgan
GameObject and its initial state, and a gesture list. This class’s purpose is to contain the
maininformationneededtobeaccessedfromanyotherscript.Furthermore,itisinitializing
thegesturelistandtheorganinitialstate.
UIManager: As it name is suggests, it manages the User Interface. At this moment, it
displaysthefeedbackicons,whichrepresenttheactivatedfeature.
Gesture:Thisclassisaparentclassforeverygesturethatwillbeimplemented.Itcontains
themainspecificationeachgestureshares.
Rotation,Translation,Zoom:Theseclassescontainthegesturesandexecutefeatures.They
areassociatedwiththeorganGameObjectascomponents.
ButtonManager: This class handles the behavior of a button as it is a 3D element in this
application.Itmanagesitsmovementandcallsanactivationmethodifthebuttonispushed.
ResetButton: This class inherits from the ButtonManager to share its behavior. The
Activationmethodcalls theResetOrganmethodof theSingleton to reset theorgan to its
initialstate.
ButtonVisibilityManager: This class is associated with the parent object of the moving
button.Thisparenthasacollidertodetect ifthehandis infrontofthebutton.Whenthe
handentersthecollider,thebuttonappearsanditcanbepushed.Thebuttondisappearsif
thehandexitsthecollider.
4.3. Resultsofthesurvey
Inthispartthefindingsandresultsoftheusersurveywillbepresented.
Intheupcomingpartsthetermaveragewillbeused.Thisvalueiscalculatedlikethis:
37
AverageTime:
𝑥 = 𝑥#
$
#
∗1𝑛
x̅istheaveragetimetakentoperformacertaintaskbyallparticipants.
x_iisthetimetakentoperformacertaintaskbyoneparticipant.
nisthenumberofparticipantstakingpartintheusersurvey.
4.3.1. GeneralInformation
The project group wants to give a brief overview over the profile of the people, who
participatedintheusersurvey.21peopleparticipatedintheusersurvey.
Figure29:Genderdemographicoftheparticipants
7participantsweremale
13participantswerefemale
1participantdidnotwanttogiveinformationaboutitsgender
38
Figure30:Agedemographicoftheparticipants
17participantswerebetween18and25yearsold
4participantswerebetween26and40yearsold
0participantswereolderthan40
Figure31:Frequencyoftheparticipantsplayingvideogames
0participantswereplayingvideogameseveryday
2participantswereplayingvideogames3-4timesaweek
3participantswereplayingvideogamesonceaweek
39
4participantswereplayingvideogamesafewtimesperyear
12participantsarenotplayingvideogamesatall
Figure32:Listofvideogames,thatareplayedbytheparticipants,whoplayvideogames
Figure33:Amountofpeopleplaying2Dor3Dvideogames
Figure34:experiencewithLeapMotionorVirtualReality.
40
19participantshaveneverworkedwithLeapMotionorVirtualReality
2participantshaveworkedwithLeapMotionorVirtualReality
Figure35:experienceinmanipulating3Dobjectsofparticipants.
3participantswereexperiencedinmanipulating3Dmodelsontheircomputer
18participantswerenotexperiencedinmanipulating3Dmodelsontheircomputer
4.3.2. Results
Thefollowingpartwillbeaboutthetakentimetosolveeachtask,theparticipants’opinion
aboutthefeaturesandabriefanalysisonhowataskwassolved.
Before assessing the gathered data, a few abnormalities concerning the execution of the
surveytasksneedtobementioned:
Nearly every participant experienced the LeapMotion for the first time. By having three
minutes of practice, participants could experience the strengths and weaknesses of the
device,butitwasnotenoughtimetoexperiencehowtodealwiththe“designproblems”,
thattheLeapMotiondevicehas.
The detection area of the LeapMotionDevice is limited, therefore itwas hard for some
participantstofindtherightposition,wheretheycouldworkwiththeLeapMotiondevice.
41
Some participants thought one could accelerate the dice by tossing it away. The survey
shouldhaveasked, iftheparticipant is leftorrighthanded,sothepositioningoftheLeap
Motion device could have been adapted. The majority of participants were missing a
physical feedback upon touching the dice or pressing the button. By taking the time, the
participantneededtosolveacertaintask,timepressurewasappliedtoparticipants.
4.3.2.1. Translation
The following figure will show how much time the participants needed to solve the
translationtask.
Figure36:AveragetimestakenfortheTranslationtask
Theparticipantsanswers,tothestatementsaboutthetranslationfeature,areshowninthe
upcomingpart.
42
Figure37:Answerstostatementoneofusersurvey
13outof21peopledisagreedwiththestatement,thatthetranslationfeatureisdifficultto
use.
Figure38:Answerstostatementtwoofusersurvey
18outof21participantsagreedwiththisstatement,whichsuggests,thattheparticipants
weresatisfiedwiththegesture,chosenfortranslatinga3Dobject.
Abnormalitiesduringthetranslationtask
Some did not knowwhat theword “Translation”meant. A few participants lost the dice
because theyweremoving their hand too fast or leaving the detection area of the Leap
43
Motiondevicebytryingtotouchthecorners.Someparticipantsdidnotpinch inthedice,
butmadeafisttotranslateit.Mostofthetimesthetranslationstillworked
4.3.2.2. Rotation
Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvetherotation
task.
Figure39:Averagetimestakenfortherotationtask
Theparticipantsanswers, to the statementsabout the rotation feature, are shown in the
upcomingpart.
44
Figure40:Answerstorotationfeaturestatement
Sevenparticipantsdisagreedwiththestatement,thattherotationfeatureistoodifficultto
use.Eightparticipantsagreedwiththatstatement.Byobservingtheparticipantsperforming
thetask,theprojectgroupfoundout,thatthegesturesfortherotationandthetranslation
featuremightbe too similar. Sinceone can rotate theobjectwith the translation feature
too.
Figure41:Answerstorotationgesturestatement
Although the gesture used for the rotationmight be similar to the gesture used for the
translation,themajorityoftheparticipantsweresatisfiedwiththegesturechosenforthe
rotationfeature.
Abnormalitiesduringtherotationtask
45
Participants pinched in the dice,whatmeans, that they grabbed the dice to rotate. They
werenotsupposedtograbthedice.Theparticipantswerenotinterrupted,andtheycould
solvethetaskinthiswayaswell.Afewparticipantshadtheissue,thatthedicemovedaway
bymistake,whiletheytriedtorotateit.
4.3.2.3. Zooming
Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvethezooming
task.
Figure42:AveragetimestakenfortheZoomingtask
Theparticipantsanswers, to thestatementsabout thezooming feature,areshown in the
upcomingpart.
46
Figure43:Answerstozoomfeaturestatement
Theprojectgroupcouldsee,thatthezoomingfeatureisdifficulttousefornearlyhalfthe
participants.
Figure44:Answerstozoominggesturestatement
Thechosengestureseemstobeacceptedbythemajorityofparticipants.Itisincomparison
totheothergesturestheonlyone,whichrequirestwohands.
AbnormalitiesduringtheZoomingTask
Occasionallythezoomingtaskwassolvedswift,becausethediceglitchedoutandchanged
itsscaletotherightscale,inwhichtheparticipantscouldreadwhatwaswrittenonthedice.
Whenthedice’sscalewasincreasedtoomuch,itfilledthewholescenery.Inthiscaseitwas
hardtodecreasethescaleofthediceagainandsometimestheprogramhadtoberestarted
byonememberoftheprojectgroup.
47
4.3.2.4. Button
Thefollowingfigurewillshowhowmuchtimetheparticipantsneededtosolvethebutton
task.
Figure45:AveragetimestakenfortheZoomingtask
Theparticipantsanswers,tothestatementsaboutthebutton,areshownintheupcoming
part.
Figure46:Answerstobuttoneaseofusestatement
48
Mostoftheparticipantsthought,thatthebuttonwaseasytopress.
Figure47:Answerstoalternativebuttonstatement
Thechosenwayofpushingthebuttonseemsfineaswell.
Abnormalitiesduringthebuttontask
A great number of participants did not have a feeling for the deepness of the scenery.
Therefore,theytriedtopushthebuttoninaplace,whereitwasnot.Peopletriedtopush
thebuttonfromtheside.Thebuttongotstucksometimes.
4.3.2.5. Additionalstatements
Figure48:AnswerstoeaseofuseofLeapMotionstatement
49
Themajorityoftheparticipantswereunsureortendedtoslightlyagree,thatLeapMotionis
easytouse.Thatisunderstandablebecauseitseems,thatLeapMotionincombinationwith
thewrittenprogramrequiressomepracticetomanipulatea3Dmodel.
Figure49:AnswerstoLeapMotionfrustrationstatement
Although Leap Motion requires some experience, most of the participants were not
frustrated by using the LeapMotion. The goal of the surveywas not to show a fun and
innovative technology, although a lot of participants stated, that they had fun doing the
surveyandthatLeapMotionisaninterestingtechnology.
Figure50:AnswerstocomfortableuseofLeapMotionstatement
OnlyonepersonfeltslightlyuncomfortableusingLeapMotion.
50
Figure51:Answerstosupportstatement
15participants said, that theywouldnotneed thehelpof a technician tobeable touse
Leap Motion. That shows the project group, that Leap Motion in combination with the
writtenprogramisintuitiveenoughinitscurrentstate.
Figure52:Answerstosurgerystatement
Nearly every participant mentioned, that this is the statement, which is the hardest to
answer. The reason for that might be, that the participants saw, that the program was
intuitive and goodworking but yet not finished enough to use it during a surgery. Some
peoplethought,thatasurgeonwoulduseLeapMotiontocontrolamachine,whichwould
performsurgeryonthem.Intheendninepeopletendtogoundersurgery,whileasurgeon
51
isusingLeapMotionand8tendtonotgoundersurgerywithasurgeonusingatechnology
likethis.
5. Discussions
5.1. SurveyAnalysis
Theusersurveywasanecessity.ItsaimwastoseehowtheparticipantsinteractwithLeap
Motion in combination with the developed program and their opinions about it. The
gathereddatawasvitalforassessingandimprovingtheprogram.Inretrospect,certainparts
of the user survey should have been changed or rethought. The following paragraph
includesthoughtstheprojectgrouphadduringanalyzingthecollecteddata.
Anumberof21participantsseemssufficient,althoughtheamountshouldhavebeenhigher
insomecases.ThepeoplewhoalreadyhadexperiencewithLeapMotionorexperiencein
manipulationof3Dmodelswerefasterineverytaskthanpeoplewithoutexperienceinany
ofthesefields.Sincethreeparticipantswereexperiencedinmanipulationof3Dmodelsand
two participants had experience with Leap Motion, the project group cannot make a
statisticallyproofedstatement,thatsaysthatthesepeoplearedefinitelyfasterthanpeople
without experience in any of these fields. The project group can state, that people,who
haveexperienceinmanipulationof3DmodelsorexperienceinworkingwithLeapMotion
tendtobefasterthanpeoplewithoutanyexperienceinthesefields.
Since some participants solved taskswithmere luck (glitching out of the dice) and some
participantshad“badluck”tryingtosolvethetasks,itisobsoletetodoastandarddeviation
oftheaveragecalculation.Asolutiontothisissuecouldhavebeentomaketheparticipants
solvethetasksmorethanonetime.Atablewiththestandarddeviationscanstillbefound
intheappendix.Thegroupdecided,thatitismoreusefultousetheMedianinstead.
Thetakentimeoftaskswasdifficulttomeasure. Insomecases,thetimeshouldnothave
beenusedtocalculateanaveragebecauseitwasatimeinfluencedbyluckorbadluck.That
isalsothereason,whycalculatingamedianwasimportant.Theprojectgroupwouldeven
gofurtherandsaythatonecanmakebetterstatementbasedonthemedianthanonthe
average. It was alreadymentioned, that doing a task several times could have fixed this
52
issue. A side effect of this solution would be, that the time of the survey would have
increasedsignificantly.
5.2. FeatureAnalysis
Allprojectmembersdevelopedtheprogram,thustheyhadsufficientpracticewiththeLeap
Motion controller tomanipulate3Dobjects throughout the semester.Most Interviewees,
whoparticipated in theuser survey,werenot familiarusing thiskindof inputdevice ina
virtual3Dspace.Therefore,theprojectteamdiscoveredcriticalissuesoftheprogramfrom
theseinterviewees,whichwerenotobviousfortheteam.
5.2.1. Grabandmove
Accordingtotheusersurvey,themajorityoftheintervieweesfoundthatthegrabandmove
feature (translation)wasnotdifficult touse,sincethegestureused for translating the3D
object was intuitive for more than 85 % interviewees. There were several complaints
concerningthemissingphysicalfeedback,whentouchingthedicebytheparticipants.Given
thiscircumstance,itissafetosay,thatnearlyallintervieweesareaccustomedtoreceiving
physicalfeedbackupontouchingatouchscreenorotherobjects.Fromaclinicalperspective,
thisargumenthas littleornovalidity.Theproject’s targetgroup,whicharesurgeons,can
use the software in combinationwith the LeapMotiondevicedirectlywithnohelpof an
assistantornurse,duetoitssterility.
5.2.2. Rotation
As stated in the results of the user survey, the rotation feature was for at least seven
participantsdifficulttouse.Theprojectteamintentionallydidnottelltheintervieweeshow
toexecutethesefeatures,inordertoseeifitwaspossibletoswiftlylearnthegesturesused
totriggerthefeatures.Thegrabandmovefeaturealsoenablesonetorotateanobject.Due
to this pre-knowledge, several interviewees successfully accomplished the rotation task
withoutusing the rotation feature.Considering this, it isessential toexplain theproject’s
targetgrouphowtoexecutecertainfeaturesandgestures.Thisfeaturerequiresnogesture,
thusitwasforthemajorityoftheparticipantsintuitive.Asreportedbythefindingsofthe
usersurvey,fewintervieweesmovedthedicewhiletryingtoexecutetherotationfeature.
53
Thisisbecausetheirhandswerenotflattenedout,thustriggeringapossibleapinch,which
resultsinactivatingthegrabandmovefeature.
5.2.3. Zoom
Intervieweesat large,seemedtothinkthegestureusedforzoomingwasintuitive,butdid
notagreeupona straightforwardexecutionof the feature.Apossible justification for the
difficulties, when scaling a 3D object in the program, might be the requirement of two
hands.Afewparticipantsdidenlargethedicetoomuch,becausetheirhandswereoutof
theLeapMotiondevicesframe.
5.2.4. Button
Themainconcernaboutthebuttonforseveralintervieweeswasthattheyhadissueswith
perceivingthelocationofthebuttoninavirtual3Dspace.Asreportedintheusersurvey,
participantsdidnothaveafeelingforthedeepnessofthescenery.Otherstriedtopressthe
buttonfromtheside,pushing itnotona linearangle.Consequently, thebuttongotstuck
severaltimes.Toresolveorminimizethisissue,itissuggestedtoanglethebuttontowards
themiddleofthescene,wheretheuserispositioned.
5.3. Suggestions
In this paragraph, there will be various ideas and optimizations explained, that might
improve this project, however they could not be implemented, due to a lack of time. It
seems, that the zoom gesture usually performs accurately while zooming the object.
However,theLeapMotiondoesrarelynotdifferentiatebetweenzoomingandtranslation.
Furthermore, theuseof thepalm to rotateanobject isnot themost intuitivemethod. It
appearedthatsomeuserscouldnotmanipulatetheobjectasfluentlyastheteamhopedto.
Forthisreason,thefollowingimprovementsshouldbecarriedoutwithspecialattentionto
thesefeatures.
Translation: tograb theobjectbypinchingwasanacceptable idea.However,whenusers
turned theirhands faceup, the virtualhandof theprogramsimply couldnotunderstand
that position, because the pinchwill be obscured by the outside of the hand. Thus, it is
recommended to implement a more intuitive gesture. One that mimics grabbing a real
54
object with one hand, by using all five fingers for wrapping around the object and thus,
grabbingtheobject.
Rotation:theuseofthepalmofthehandwaslessintuitive,fewparticipantsthoughttoput
their palm inside the object and then start rotating the object. For this reason, it would
seemlogical tohavethesamehandgesturethetranslationpossesses.Thatwouldreduce
thelearningcurve(timeyouneedtogainnewskills).
Button:attherightpartofthescreenthereisabuttonwhichallowsuserstoresettheinitial
positionof theobject.While therearenosignificantcomplaintsor issueswhenusing this
feature,itwasseenthatwhenlefthanderstriedtopressittheyhadmoredifficultiesthan
thosewhoarerighthanders.So,beforestartingtheprogramthereshouldbeanoptionthat
asksyouwhichyourdominanthandis.
5.4. Generalanalysis
5.4.1. Relevance
Thisprogramhasbeendeveloped inorder to facilitate the interactionof3Dobjects. This
projectcanbringlastingchangesinthemedicalcommunity.Forinstance,partsofthecode
couldperfectlybeusedinotherprogramsorapplications.
5.4.2. Effectiveness
Theobjectiveswerechangedasthesemesterprogressed.Firstly,theobjectivewastoadd
fewadditional features to theprevious code,however theprojectgroupdidnot improve
uponthepriorproject’scode,duetocompatibilityissuesondifferentoperatingsystems.At
thatmoment, objectiveswere drasticallymodified in order to reach the final goal,which
allows surgeons to control three dimensional models while they are working. Another
challenging goal was to develop a program that might be enhanced by future students.
Furthermore, sometimes itwasdifficult toarrangemeetingswith thegroupmembers.At
theend, theproject could achievealmost all themain goals, suchasdevelopaprogram,
which users could manipulate 3D objects this involves translate them, rotate them and
zoomin/outthemandalso,wasabletorunonMacOSandWindows.Theteamrequired
55
additionaltime,inordertocompletetheoptimizationspreviouslymentioned,resultingina
betterinteractionbetweentheusersandtheprogram.
5.4.3. Efficiency
AGanttdiagramwasused tocoordinate tasks throughout thesemester, so thatonegoal
canbedividedinvariousobjectives.Subsequently,objectivesweresplitintotasksforteam
memberstofinishinacertaintimeframe.Tosumup,objectiveswereachievedontime.
5.4.4. Sustainability
Willtherebeanexitstrategyfortheproject?Theremightbeone,astheprogramisworking
wellandmostofthefunctionalitiesareeasytouseevenifoneisnotanadvanceduserin
manipulating3DmodelswithLeapMotion.Exitstrategiesfortheproject:
● Firstly,theprojectmightbeenhancedbythenextEPSstudents.
● Secondly,thisprojectcouldcontributetoimproveopensourceworld,beingusedin
futureapplications,bylettingpeopledownloaditforfree.
● Last one, the program could be sold to a private company as long as they were
interestedinitpreviously.
56
6. Conclusion
In conclusion, a prototype with basic features has been developed with the aid of a
conducted user survey. Additionally, a documentation of the code has been written for
futuredevelopment.
Aftera careful inspectionof thepriorproject’sprogram,and its results, theproject team
agreedupondevelopinganewprogramforthesakeofinstantgesturerecognitionandthe
supportofcross-platformcompatibility.
The group implemented various features, such as rotating, scaling and translating a 3D
object.Additionally,gesturesforexecutingthesefeatureshavebeendeveloped,providing
the possibility to select a feature without physically pressing a button on a keyboard or
computermouse.
Aquantitativemethodwasusedtogetuserdata,andasurveywasconductedamongthe
users.Thegathereddatamadeitpossibletosetupstatisticsandformulatesuggestionsfor
evaluationandassessmentof theprogram.Beside that, theproject teamdiscoveredhow
onemightutilizetheLeapMotiondevicefortheirfirsttimemanipulating3Dobjects.
Intheend,theprojectteamdevelopedaprototypeforsurgeonstomanipulate3Dobjects
directly in amore intuitive and comfortablemanner. Instead of using a computermouse
indirectly,theynowcouldutilizehandgesturestomanipulate3Dobjects.Duetothelackof
time, further research and development could have been made. Hence, the project
publishedthesoftwareonGitHubforfuturedevelopment.LeapMotionhasthepotentialto
enhance the human-computer interaction in a clinical environment. It is capable to
revolutionize medical 3D and 2D imaging, due to its sterility, ease of use and low-cost
barrier.
57
7. PersonalReflection(Belbin)
LeapMotion is a fascinating technology. Aswewere looking through all the projects the
European Project Semester had to offer, wewere all thrilled by the idea of this subject.
Cooperatingwithahospital to improve surgeryusinghigh technologywasanexceptional
opportunityforus.Thisprojectgaveusthechancetoworkonarealproductthatmightlead
toasubstantialimpactforthefutureofsurgery.Therefore,itgrantedusmotivationtostart
theproject.
Although,wetriedtotesttheapplicationoftheformergroupandunderstoodthatitwould
notbepossibletocontinuetheirwork.Achoicehadtobemadebetweentwoversionsof
theLeapMotion.Wedidnotwanttorevertourworkduringthesemester,thusittookusa
longtimetodecidewhichsolutionwillbethemoststable.
Whenourchoiceofstartingfromscratchwiththeolderversionwasmade,wecommenced
programming. The coding levels of the teamwere very heterogeneous. Hence, the tasks
weredivideddifferentlyandtimehadtobegivenfor the leastexperienceddevelopersto
helpthem.Thegoodpointbehindthisisthattheyhavenowabetterunderstandingofthe
developer'swayofworkingandhaveimprovedtheirskillswithprogrammingtoolssuchas
GitHub,UnityandStackOverflow.
Asanalphaversionwasdeveloped,wechosetodividethegroupintwosubgroups.Thetwo
developerskeptcodingontheapplicationforoptimizationwhilethetwoothersweredoing
theusersurvey.Thiswasanexcellent ideabecauseeveryoneworkedmoreefficientlyand
the application was finished in no time. Besides, the user survey gave us a real
understandingof theLeapMotiondeviceandour software.Wealso learneda loton the
methods of surveying and it gave us a first experience on this matter for our upcoming
projects.
Attheend,theprojectachieveditsgoals.Weareproudofwhatwehavedoneandhopeit
willcontinueinthefuturewithanEPSgrouporanyteamofdevelopers.
Thissuccessisnotonlyduetoourselves,butalsotoLouiseOramoursupervisor.Shehelped
uswithvarioustasksandgrantedusconstructivefeedbacksduringtheentireproject.Thus,
58
herexpertiseguidedustowardsaccomplishablegoals.Inaddition,shewasthecontactwith
the hospital and consultant companies that could help us. She gave us a one-time
opportunity to assists a real surgery and invited us to a meeting with Sopra Steria, a
consultancyworkingonvirtualtechnologies.
Inamorepersonalway,our teamhadanoutstandingtimeatHiOA;workingontheLeap
Motionprojectbutmostlyworkingwithexceptionalinternationalstudents.Everyoneinour
group had a different way of working or even seeing the world. Communication was
sometimedifficultandleadtoanargument.However,wealwaysrespectedeachotherand
thedebatealways finishedwithanagreement.Besides,wediscusseda lot tounderstand
eachother,weacceptedeachother’sdifferencesandwetookadvantageofthem.Different
opinionsgivedifferentapproaches,itopenedourfieldofvision.Moreover,aninternational
groupmeansamultitudeofpersonalities.Asyoucanseeinthechartbelow,weweretotally
differentcharacters.Andwealsohaddistinctivebackground,suchasthefieldofstudyor
even the education. We have benefited a lot from this diversity. The most experienced
developerschosetoleadthegroupashewasalsoagoodcoordinatorandshaper.Hence,
he could give the other clear tasks and goals that simplified the workflow dramatically.
Nevertheless,theothermembersofthegrouphaddistinctqualitiesthatimprovedtheteam
work. Some used their technical skills to develop the application. Some used their
communicationskillstorealizetheusersurvey.Finally,someusedtheirwritingskillstohave
aclearacademicreport.
59
Figure53:Belbinresultsofthefourmembersofthegroup
To concludewith, this EPS project taught us several lessons.We learnedwhatwas Leap
Motion and saw that it was not as advanced as it looks like. Some improved their
programmingskillsandtheotherstheirteachingskills.Furthermore,wemainlylearnedhow
to work in an international team, listening to everyone’s opinion, expressing ours and
makingcompromises.Finally,wehadtospeakingEnglish,whenthemomentwasjoyfulor
whenitwasstressful,whenwefeltexcitedorwhenwewereexhausted.Asnomemberof
the team was from an English-speaking country, we all had to do this effort. It was
sometimeschallengingbutweallfaceditandovercameit.Thisprojectwasuniqueandwill
leadustomanyotherinternationalcollaborationsinthefuture.
60
8. References
Balakrishnan,R.,Baudel,T.,Kurtenbach,G.,&Fitzmaurice,G.(1997).TheRockin'Mouse.
ProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems-CHI
'97.doi:10.1145/258549.258778
Noller,F.(2016,July31).Gettingstartedwithsimple3DButtonsusingUnityandLeap
Motion.Retrievedfromhttps://medium.com/@felixnoller/getting-started-with-
simple-3d-buttons-using-unity-and-leap-motion-40d9efded317
61
9. Illustrationdirectory
Figure1:handpushinga3Dbutton____________________________________________16
Figure2:GeneralInformationabouttheLeapMotionProject_______________________20
Figure3:Startofthecollectionofpersonalinformation.Askingaboutthegender.______21
Figure4:Askingabouttheage. _______________________________________________21
Figure5:Askingabouthowoftenthesurveyparticipantisplayingvideogames.________22
Figure6:Askingaboutwhichvideogamestheparticipantplays._____________________22
Figure7:Askinginhowmanydimensionsthevideogamesareplayed.________________23
Figure8:AskingabouthavingexperiencewithLeapMotionorVirtualReality.__________23
Figure9:Askingiftheparticipanthasexperienceinmanipulating3Dmodels.__________23
Figure10:Virtualenvironment,inwhichtheparticipantofthesurveyinteracts.________24
Figure11:IntroductionbeforetheLeapMotionprogramwasdemonstrated. __________25
Figure12:Introducingtheupcomingtasks.______________________________________25
Figure13:ExplanationoftheTranslationTask ___________________________________26
Figure14:movementprocedure______________________________________________27
Figure15:Explanationoftherotationtask______________________________________27
Figure16:Rotationofthedice________________________________________________28
Figure17:ExplanationoftheZoomingTask _____________________________________28
Figure18:Diceafterchangingitsscaling. _______________________________________29
Figure19:ExplanationoftheButtonTask_______________________________________29
Figure20:Pressingthebutton________________________________________________30
Figure21:Explanationofthereviewingsystem __________________________________30
Figure22:FirsttwostatementsabouttheTranslationfeature.______________________31
Figure23:StatementsabouttheRotationfeature ________________________________31
Figure24:StatementsabouttheZoomingfeature________________________________32
Figure25:StatementsabouttheButtonfeature__________________________________32
Figure26:StatementsaboutLeapMotioningeneral______________________________33
Figure27:AdditionalStatementsandendofthesurveybypressingthesubmitbutton.__33
Figure28:ObjectDiagramoftheApplication ____________________________________35
Figure29:Genderdemographicoftheparticipants _______________________________37
62
Figure30:Agedemographicoftheparticipants__________________________________38
Figure31:Frequencyoftheparticipantsplayingvideogames_______________________38
Figure32:Listofvideogames,thatareplayedbytheparticipants,whoplayvideogames 39
Figure33:Amountofpeopleplaying2Dor3Dvideogames_________________________39
Figure34:experiencewithLeapMotionorVirtualReality. _________________________39
Figure35:experienceinmanipulating3Dobjectsofparticipants.____________________40
Figure36:AveragetimestakenfortheTranslationtask____________________________41
Figure37:Answerstostatementoneofusersurvey ______________________________42
Figure38:Answerstostatementtwoofusersurvey ______________________________42
Figure39:Averagetimestakenfortherotationtask ______________________________43
Figure40:Answerstorotationfeaturestatement ________________________________44
Figure41:Answerstorotationgesturestatement ________________________________44
Figure42:AveragetimestakenfortheZoomingtask______________________________45
Figure43:Answerstozoomfeaturestatement___________________________________46
Figure44:Answerstozoominggesturestatement________________________________46
Figure45:AveragetimestakenfortheZoomingtask______________________________47
Figure46:Answerstobuttoneaseofusestatement ______________________________47
Figure47:Answerstoalternativebuttonstatement_______________________________48
Figure48:AnswerstoeaseofuseofLeapMotionstatement _______________________48
Figure49:AnswerstoLeapMotionfrustrationstatement__________________________49
Figure50:AnswerstocomfortableuseofLeapMotionstatement ___________________49
Figure51:Answerstosupportstatement_______________________________________50
Figure52:Answerstosurgerystatement _______________________________________50
Figure53:Belbinresultsofthefourmembersofthegroup_________________________59
63
AppendixA:SourceCode
Loader.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassLoader:MonoBehaviour{ voidAwake(){ Singleton.Instance.Init(); gameObject.AddComponent<UIManager>(); Singleton.Instance.UIMgr=gameObject.GetComponent<UIManager>(); }}
64
ButtonManager.csusingSystem;usingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;usingLeap;publicclassButtonManager:MonoBehaviour{ /************** *Attributes* ***************/ publicboollockX; publicboollockY; publicboollockZ; publicfloatreturnSpeed; publicfloatactivationDistance; publicColorinactiveColor; publicColoractiveColor; protectedboolpressed=false; protectedboolactivated=false; publicboolfading=false; protectedVector3startPosition; /*********** *Methods* ************/ //Usethisforinitialization voidStart(){ //Rememberstartpositionofbutton startPosition=transform.localPosition; gameObject.SetActive(false); } //Updateiscalledonceperframe voidUpdate(){
65
floatinitPos=0.0f; if(!lockX)initPos=startPosition.x; elseif(!lockY)initPos=startPosition.y; elseif(!lockZ)initPos=startPosition.z; floatoffset=(activationDistance-initPos)*0.2f; if(transform.localPosition.z>offset) moveButton(); else transform.localPosition=startPosition; if(activated){ activated=false; Activation(); } } protectedvoidmoveButton() { //Uselocalpositioninsteadofglobal,sobuttoncanberotatedinanydirection //Lockdisabledaxis Vector3localPos=transform.localPosition; if(lockX)localPos.x=startPosition.x; if(lockY)localPos.y=startPosition.y; if(lockZ)localPos.z=startPosition.z; transform.localPosition=localPos; //Getdistanceofbuttonpress.Makesuretoonlyhaveonemovingaxis. floatdistance=GetDistance(); //ReturnbuttontostartPosition transform.localPosition=Vector3.Lerp(transform.localPosition,startPosition,Time.deltaTime*returnSpeed); floatpressCompletion=1.0f-((activationDistance-distance)/activationDistance); //buttonpressed if(pressCompletion>=0.7f&&!pressed) { pressed=true; activated=true; //Changecolorofobjecttoactivationcolor StartCoroutine(ChangeButtonColor(gameObject,inactiveColor,activeColor,0.2f)); }
66
//buttonunpressed elseif(pressCompletion<=0.30f&&pressed) { pressed=false; //Changecolorofobjectbacktonormal StartCoroutine(ChangeButtonColor(gameObject,activeColor,inactiveColor,0.3f)); } } protectedIEnumeratorChangeButtonColor(GameObjectobj,Colorfrom,Colorto,floatduration){ floattimeElapsed=0.0f; floatt=0.0f; while(t<1.0f) { timeElapsed+=Time.deltaTime; t=timeElapsed/duration; obj.GetComponent<Renderer>().material.color=Color.Lerp(from,to,t); yieldreturnnull; } } //thebuttonfadesawayanddeactivatesitself protectedIEnumeratorFadeAway(GameObjectobj,floatduration){ fading=true; floattimeElapsed=0.0f; floatt=0.0f; while(t<1.0f) { timeElapsed+=Time.deltaTime; t=timeElapsed/duration; ColorcolorAlpha=obj.GetComponent<Renderer>().material.color; colorAlpha.a=Mathf.Lerp(1.0f,0.0f,t); obj.GetComponent<Renderer>().material.color=colorAlpha; yieldreturnnull; } //Forthefadeawayfunction,deactivatesgameobjectwhentransparent if(gameObject.GetComponent<Renderer>().material.color.a<0.05f){ fading=false;
67
gameObject.SetActive(false); } } //togetthedistancebetweenthebuttonandit'sstartposition protectedfloatGetDistance(){ Vector3allDistances=transform.localPosition-startPosition; floatdist=1.0f; if(!lockX)dist=Math.Abs(allDistances.x); elseif(!lockY)dist=Math.Abs(allDistances.y); elseif(!lockZ)dist=Math.Abs(allDistances.z); returndist; } //actionofthebuttonwhenpressed,fortheButtonManageritischangecolor publicvirtualvoidActivation(){ floatr,g,b; r=UnityEngine.Random.Range(0.0f,1.0f); g=UnityEngine.Random.Range(0.0f,1.0f); b=UnityEngine.Random.Range(0.0f,1.0f); Singleton.Instance.Organ.GetComponent<Renderer>().material.color=newColor(r,g,b); } //thebuttondisappears:fadesaway publicvoidDeactivation(){ if(fading==false){ StartCoroutine(FadeAway(gameObject,0.5f)); } }}
68
ButtonVisibilityManager.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassButtonVisibilityManager:MonoBehaviour{ protectedGameObjectmovingButton; //Usethisforinitialization voidStart(){ movingButton=transform.Find("MovingButton").gameObject; } //Updateiscalledonceperframe voidUpdate(){ } voidOnTriggerEnter(ColliderotherCollider){ if(otherCollider.transform.parent.name=="middle"&&otherCollider.gameObject.name=="bone1"){ movingButton.SetActive(true); } } voidOnTriggerExit(ColliderotherCollider){ if(otherCollider.transform.parent.name=="middle"&&otherCollider.gameObject.name=="bone1"&&movingButton.activeInHierarchy){ movingButton.GetComponent<ButtonManager>().Deactivation(); } }}
69
Rotation.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassRotation:Gesture{ Controllercontroller; publicHandControllerhandCtrl; privateQuaternionoldHandRotation; //Usethisforinitialization voidStart(){ controller=newController(); oldHandRotation=Quaternion.identity; } privateboolIsHand(Colliderother){ if(other.transform.parent&&other.transform.parent.parent&&other.transform.parent.parent.GetComponent<HandModel>()) returntrue; else returnfalse; } voidOnTriggerEnter(){ oldHandRotation=Quaternion.identity; } voidOnTriggerStay(Colliderother){ if(IsHand(other)) { Frameframe=controller.Frame(); Handhand=frame.Hands.Frontmost; QuaternionnewHandRotation=handCtrl.transform.rotation*hand.Basis.Rotation(false); if(oldHandRotation!=Quaternion.identity){ QuaterniondeltaRotation=newHandRotation*Quaternion.Inverse(oldHandRotation); transform.rotation=deltaRotation*transform.rotation; //transform.rotation=handCtrl.transform.rotation*
70
hand.Basis.Rotation(false);//rotation Activated=true; } oldHandRotation=handCtrl.transform.rotation*hand.Basis.Rotation(false); } } voidOnTriggerExit(){ Activated=false; oldHandRotation=Quaternion.identity; }}
71
Translation.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassTranslation:Gesture{ Controllercontroller; booltrigger_inside=false; publicbooltrigger_insideL=false; publicbooltrigger_insideR=false; publicHandControllerhandCtrl; publicfloatoldZoomdistance; publicboolisLeftFirst=false; publicboolisLeftSecond=false; publicboolzooming=false; publicbooltranslating=false; publicbooltrigger_pinch1=false; publicbooltrigger_pinch2=false; publicVector3thumb_tip1; publicVector3thumb_tip2; voidStart() { controller=newController(); } voidUpdate(){ Frameframe=controller.Frame(); HandListhandsInFrame=frame.Hands; Handhand=frame.Hands.Frontmost; trigger_pinch1=false; trigger_pinch2=false; Fingerfinger=newFinger(frame.Pointables.Frontmost); floatTHUMB_TRIGGER_DISTANCE=0.025f; HandfirstHand=handsInFrame[0]; HandsecondHand=handsInFrame[1]; thumb_tip1=firstHand.Fingers[0].TipPosition.ToUnityScaled(); thumb_tip2=secondHand.Fingers[0].TipPosition.ToUnityScaled(); isLeftFirst=firstHand.IsLeft?true:false; isLeftSecond=secondHand.IsLeft?true:false; //checkingifhandspinch for(inti=1;i<firstHand.Fingers.Count&&!trigger_pinch1;++i){ Fingerfinger2=firstHand.Fingers[i];
72
for(intj=0;j<22&&!trigger_pinch1;++j){ Vector3joint_position=finger2.JointPosition((Finger.FingerJoint)(j)).ToUnityScaled(); Vector3distance=thumb_tip1-joint_position; if(distance.magnitude<THUMB_TRIGGER_DISTANCE) trigger_pinch1=true; } } for(inti=1;i<secondHand.Fingers.Count&&!trigger_pinch2;++i){ Fingerfinger3=secondHand.Fingers[i]; for(intj=0;j<22&&!trigger_pinch2;++j){ Vector3joint_position2=finger3.JointPosition((Finger.FingerJoint)(j)).ToUnityScaled(); Vector3distance2=thumb_tip2-joint_position2; if(distance2.magnitude<THUMB_TRIGGER_DISTANCE) trigger_pinch2=true; } } //executingthemovement if(handsInFrame.Count>0){ if(finger!=Finger.Invalid){ //translationfunction-ifhandisinsideandpinched->youcanmovetheobject if((trigger_pinch1||trigger_pinch2)&&trigger_inside&&!zooming){ Activated=true; //usedeltaposition transform.position=handCtrl.transform.TransformPoint(hand.Fingers[0].TipPosition.ToUnityScaled());//translation }else{ trigger_inside=false; trigger_insideL=false; trigger_insideR=false; } } } } //checkingifhandsarecollidingwiththeobject voidOnTriggerStay(Colliderother){
73
if(other.transform.parent.root.name=="RigidRoundHandLeft(Clone)"){ trigger_insideL=true; }elseif(other.transform.parent.root.name=="RigidRoundHandRight(Clone)"){ trigger_insideR=true; } trigger_inside=true; } voidOnTriggerExit(Colliderother){ trigger_insideL=false; trigger_insideR=false; Activated=false; translating=false; zooming=false; } }
74
Zoom.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingLeap;publicclassZoom:Gesture{ Controllercontroller; privateTranslationT; voidStart(){ controller=newController(); T=gameObject.GetComponent<Translation>(); } voidUpdate(){ Frameframe=controller.Frame(); HandListhandsInFrame=frame.Hands; booltrigger_insideR=T.trigger_insideR; booltrigger_insideL=T.trigger_insideL; booltrigger_pinch1=T.trigger_pinch1; booltrigger_pinch2=T.trigger_pinch2; boolzooming=T.zooming; floatoldZoomdistance=T.oldZoomdistance; Vector3thumb_tip1=T.thumb_tip1; Vector3thumb_tip2=T.thumb_tip2; if(handsInFrame.Count==2&&trigger_insideR&&trigger_insideL&&trigger_pinch1&&trigger_pinch2){ floatdistance3=Vector3.Distance(thumb_tip1,thumb_tip2)*50; if(oldZoomdistance==distance3){ zooming=false; }else{ zooming=true; Activated=true; transform.localScale=Vector3.one*distance3; } oldZoomdistance=distance3; }else{ Activated=false; } }}
75
UIManager.csusingSystem;usingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;usingLeap;publicclassUIManager:MonoBehaviour{ publicUnityEngine.UI.ImagegestureImage; //Usethisforinitialization voidStart(){ //InitforGesturefeedbackicon gestureImage=GameObject.Find("GestureIcon").GetComponent<UnityEngine.UI.Image>(); gestureImage.enabled=false; } publicvoidDetectGesture(){ boolimageEnabled=false; foreach(GesturegestureinSingleton.Instance.GetGestureList()){ if(gesture.Activated){ gestureImage.enabled=true; gestureImage.sprite=gesture.Icon; imageEnabled=true; //makingsuretranslatingicongetsselectediftranslatingandrotatingatthesametime. if(gesture.GetType().Equals(typeof(Rotation))){ foreach(GestureotherGestureinSingleton.Instance.GetGestureList()){ if(otherGesture.GetType().Equals(typeof(Translation))&&otherGesture.Activated){ gestureImage.sprite=otherGesture.Icon; break; } } } break; }
76
} if(imageEnabled==false){ gestureImage.enabled=false; } } voidUpdate(){ DetectGesture(); }}
77
Gesture.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;usingUnityEngine.UI;publicclassGesture:MonoBehaviour{ publicSpriteIcon; publicboolActivated; //Usethisforinitialization voidStart(){ Activated=false; } //Updateiscalledonceperframe voidUpdate(){ }}
78
ResetButton.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassResetButton:ButtonManager{ //VirtualisationofthefunctionActivation:Resetfunctionisactivated publicoverridevoidActivation(){ Singleton.Instance.ResetOrgan(); }}
79
Singleton.csusingSystem.Collections;usingSystem.Collections.Generic;usingUnityEngine;publicclassSingleton{ /************** *Attributes* ***************/ privatestaticSingletoninstance; publicUIManagerUIMgr; privateList<Gesture>gestureList; publicGameObjectOrgan; publicVector3initialOrganPosition; publicQuaternioninitialOrganRotation; publicVector3initialOrganLocalScale; /*********** *Methods* ************/ privateSingleton(){} publicstaticSingletonInstance { get { if(instance==null) { instance=newSingleton(); } returninstance; } } publicvoidInit(){ InitOrgan(); InitGestureList(); }
80
publicvoidInitOrgan(){ Organ=GameObject.FindGameObjectWithTag("Organ"); initialOrganPosition=Organ.transform.position; initialOrganRotation=Organ.transform.rotation; initialOrganLocalScale=Organ.transform.localScale; } publicvoidResetOrgan(){ Organ.transform.position=initialOrganPosition; Organ.transform.rotation=initialOrganRotation; Organ.transform.localScale=initialOrganLocalScale; } publicvoidInitGestureList(){ gestureList=newList<Gesture>(); Gesturezoom=Organ.GetComponent<Zoom>(); Gesturerotation=Organ.GetComponent<Rotation>(); Gesturetranslation=Organ.GetComponent<Translation>(); gestureList.Add(zoom); gestureList.Add(rotation); gestureList.Add(translation); } publicList<Gesture>GetGestureList(){ returngestureList; }}
81
AppendixC:InstallationManual
1. DownloadandinstallthelatestversionofUnity(2017.1.1orabove).
2. DownloadtheLeapMotionCoreAssetsv2.3.1fromGitHubandextractit.
3. DownloadtheprojectfromGitHubasaziparchiveandextractthearchive.
(https://github.com/schurlbua/LeapMotion)
4. Open the folder of the project and open the file DefaultScene.unity with Unity,
locatedin/source/LeapUnitySDK2/Assets/DefaultScene.unity.
5. InUnity navigate toAssets -> Import Package -> Custom Package… and open the
LeapMotionCoreAssetsv2.3.1.
6. Onceasmallwindowopens,clickImportandwaittillUnityhassuccessfullyinstalled
theassets.
7. Duetothemissinglinksoftheobject’sscriptsandmaterials,itisadvisedtofollow
furtherinstructionsontheproject’sGitHub.
(https://github.com/schurlbua/LeapMotion)
82
AppendixC:MethodsUserStudy
ScriptforpresentingtheLeapMotionfeatures
“Thereare4 features Iwant to showyoubeforeyouhave3minutes time topracticeon
yourownandexperienceLeapMotion”
Translation:
“NowIwillshowyouhowyoucantranslatethedice.”
Show the translation and emphasize on, that you have to pinch to grab the object. Just
translatethediceabitaround.
Rotation:
“Iwillshowyoutherotationfeaturenow.”
Showtherotation featureandmention, that theprogramdiffersbetweenhavingonly the
handinthecubeandhavingthehandinthecubeandpinching.
Zooming:
“Iwillshowyouthezoomingfunctionnow.”
Showhowyoucanzoominandzoomout.Emphasize,thatyouhavetobewithbothhands
inthecubeandpinch.
Button:
“Iwillshowyoutheuseofthebuttonnow.”
Movethediceawayandpressthebuttonafewtimes.Lateron,thebuttontaskalwayshas
tobeafterthetranslationtask.
GeneralInformation:
Mentionthatthepeopletestingtheprogramcanalwaysaskustoresettheprogram.Ifthey
askthis,wewillresettheprogram,butthetimewillkeeponrunning.
83
AppendixD:ResultsUserStudy
Table1:takentimesofindividualparticipantsinseconds.
Tasks Translation Rotation Zooming
#1 19,53 2,50 10,06
#2 19,88 18,80 15,14
#3 8,96 7,03 1,27
#4 12,72 12,49 70,62
#5 30,79 42,71 27,77
#6 22,62 5,73 11,20
#7 13,97 11,61 8,18
#8 55,36 17,14 8,31
#9 16,80 17,12 71,90
#10 45,30 17,58 8,40
#11 28,99 82,30 12,19
#12 13,76 14,25 11,06
84
#13 10,67 18,93 11,16
#14 36,84 19,05 34,53
#15 26,99 9,84 62,65
#16 14,00 7,82 10,42
#17 12,53 16,49 7,65
#18 14,43 17,58 20,71
#19 54,64 9,40 9,18
#20 39,48 5,21 12,05
#21 28,04 15,83 5,56
Table2:Averagetimesofdifferentcategoriesofparticipants
Task/Rubrik Translation Rotation Zooming Button
Average(21) 25,06 17,59 20,48 21,39
Median(21) 19,88 15,83 11,16 14,66
Boys(7) 19,12 29,14 30,99 19,34
Girls(13) 25,99 12 15,68 23,73
85
18-25(17) 24,43 14,46 19,45 18,63
26-40(4) 27,76 30,89 24,83 35,95
VG:3-4timesa
week(2)42,72 26,06 18,48 14,05
VG:Oncea
week(3)14,42 14,47 12,37 13,16
VG:Afewtimes
peryear(4)16,71 8,75 10,71 15,85
VG:notplaying
any(12)27,57 19,91 26,09 27,46
Leap/VRno(19) 26,22 17,77 20,96 22,67
Leap/VRyes(2) 14,10 15,92 15,89 14,86
3Dno(19) 25,89 18,10 20,75 22,63
3Dyes(2) 20,08 14,55 18,84 17,72
Table3:Standarddeviationsofdifferentcategoriesofparticipants
Task/Rubrik Translation Rotation Zooming Button
Average 13,81 16,56 20,88 16,35
86
Boys 7,35 23,76 26,11 20,04
Girls 13,84 5,55 15,65 14,38
18-25 12,81 8,88 19,18 11,21
26-40 17,17 29,76 26,49 25,04
3-4timesa
week
11,93 16,66 9,30 2,55
Onceaweek 4,46 5,28 8,17 1,09
Afewtimesper
year
4,66 6,18 0,49 5,41
notplayingany 13,71 19,18 25,52 19,61
Leap/VRno 14,03 17,39 21,84 17,02
Leap/VRyes 0,34 1,67 4,83 0,20
3Dno 13,90 17,70 21,84 17,41
3Dyes 12,06 5,35 13,64 5,77