abstract - faculty of engineering | imperial college · pdf filef igurate: an application for...

Abstract

Animating a c aracter b and can be a long and difficult process ecnologies suc as motion capture e ist t at can detect pose of an actor witaccurac but t ese s stems are t picall too e pensi e and difficult to set up fort e a erage user

n t is pro ect we propose a no el solution using computer ision tec ni uesto building a s stem t at can detect t e pose of a real world ob ect e s stemis eas to use and does not re uire an t ing ot er t an an ine pensi e web camto operate ome of t e uses for t e s stem include being used as an electronicdrawing aid for artists or for electronic stop motion animation

Acknowledgements

irst and foremost would like to t ank r Andrew a ison for acceptingm pro ect proposal and for is continued guidance and support t roug out t ispro ect would also like to t ank uncan illies for is inital feedback duringt e earl stages of t e pro ect inall would like to t ank m famil andfriends for t eir continued support

f igurate: An application for rigid skeleton modelling

ontents

ntroduction

b ecti e

oti ation

isting tec nologies

ssues

ontributions

ackground

omputer ision

amera model

undle Ad ustment

on linear optimisation

A

arker detection

A oolkit

A A

otion capture file formats


mplementation

ser efined odel

one model

arkers

onstraints

ile format

ideo input

ser nterface

arker pose detection

amera calibration

nitial approac es

A approac

arker distance approac

inal approac : arker pro ection from iew

b ecti e function

mplementation

aluation

uantitati e anal sis

ase model markers

ualitati e anal sis

reating t e model

stimation of model pose

i


onclusion

ummar

imitations and future work

A Appendi

A ardware

A oftware

ibliograp

ii


ntroduction

b ecti e

e ob ecti e of t is pro ect is to de elop a s stem t at utilises computer isiontec ni ues to estimate t e pose of a real world poseable model or instance we canrig a umanoid skeleton from a drawing manne uin of t e same proportions es stem will be able to determine t e pose of t e real world ob ect based on a numberof markers placed on t e ob ect

n order for t e s stem to accuratel estimate t e pose of t e model t e estimations ofpositions of t e markers must be as accurate as possible urt ermore t e s stem

must be able to pose t e model as best as it can gi en imprecise or missing informationabout t e positions of parts of t e ob ect due to noise or occlusion omputerision tec ni ues will be used to detect recognise and estimate t e location of t e

markers in space e s stem s ould be designed to be able to work wit anc eap off t e s elf web cam

sers s ould also be able to configure t eir own model definitions in order to estimatepose of custom models t at ma not be umanoid in nature e model definition


is a specification t at describes t e structure of t e model t at t e user wants to runt roug t e s stem or instance in a umanoid model it would contain informationabout t e lengt of specific bones w ic bones oin toget er t e degrees of mo ementof oints t at oin bones and t e angle of mo ement of t e oints An intuiti ewill also be created t at will allow t e user to create t is model definition

e scope of t is pro ect is to create t e abo e s stem and to make it as eas to useas possible for t e end user a full fledged will be designed and t e applications ould pro ide real time feedback t roug out t e process of estimating t e pose of t eskeleton model

oti ation

e moti ation be ind t is pro ect is to make t e act of posing animation skeletons band easier e primar uses of t is s stem is to pro ide a better drawing aid for

artists and for fast compositing of static scenes for artists

an traditional artists use image references to aid t em in drawing rawing umanoid sub ects can be especiall difficult for amateur artists raditional scaled umanmanne uins can be used to elp but t e s stem described in report impro es t is as itcan be scaled to an bod t pe and can be sa ed electronicall A ke ad antage is t ata static iewpoint of t e model can also be sa ed as an image meaning t at t e artistcan be referring to t e e act same image and iewpoint e er time is can be eruseful if t e user is using digital p oto editing tools suc as otos op w ere t e candirectl import t e final result

isting solutions are eit er too comple or e pensi e for t e ma orit of artists andanimating a skeletal rig can be a long and cumbersome process and re uires trainingand e pertise to get a good result isting motion capture s stems used to rigskeletons are t picall er e pensi e and re uire e tensi e calibration and clean up toac ie e good results

e main area of researc in t is pro ect is in computer ision ecentl t ere asbeen a lot of interest in computer ision due to t e ubi uit of cameras on mancomputing de ices suc as smart p ones and tablets as well as t e ad ances in computerprocessing power allowing man computer ision tec ni ues to run in real time akingt e computer ision approac to t e problem of rigging t e skeleton automates a lot oft e work re uired and makes it faster t an traditional approac es suc as and riggingfurt ermore it is more user friendl as t e user does not need to know a lot of t edetails in ol ed in animation


e s stem proposed in t is pro ect aims to make it eas and c eap to pose a staticskeleton using a real world counterpart suc t at an one wit out knowledge ofskeleton posing can use it

isting tec nologies

e field of pose detection as been well e plored as it is commonl used in motioncapture tec nolog otion capture can be described as t e process of recording t emo ement of an ob ect or person o er time n film making and games it is usedto record t e mo ement of bod parts in order to animate c aracters in computeranimation ewer motion capture s stems can e en capture facial e pression

raditionall motion capture s stems used markers in order to detect t e positions of t e ob ect ultiple ig speed cameras would be strategicall placed in t eperformance area and lig ting conditions would be controlled er frame of datacaptured from t e cameras would be s nced and t e positions of parts of t e bodwould be triangulated from t ese markers ese s stems t picall offer er accuratepose detection owe er t e downside is t e need for er precise calibration and set upas well as t e cost of multiple ig resolution speed cameras normall upwards ofcameras t e large space and t e software needed to run suc a s stem t can t picallcost undreds of dollars in order to build a motion capture not to take into accountt e cost of specialists needed to clean up and process t e data from t e output of t es stem

A more recent approac comes in t e form of t e icrosoft inect e inectis a low cost dept camera t at is capable of detecting pose of a umanoidsub ect wit less t an a cm error for eac bod part t works b generating apoint cloud using its dept sensor w ic it t en feeds into proprietar softwarew ic is able to make sense of t e point cloud and con ert it into a skeletal pose inless t an ms en de eloping t e inect man terab tes of data of people indifferent poses were collected and bod parts were labeled and fed into an e perts stem running on a powerful cluster of computers is resulted in a software packaget at was capable of utilising t is past data to accuratel estimate t e pose of a umanbod from an frame of point cloud data from t e sensor e software was trainedwit ot er euristics suc as ow t e bod is oined toget er for instance t at t e andis connected to t e arm in order to make it faster and more accurate e downsidesof inect is its limitation to onl work on umanoid sub ects and t at t e sub ect mustbe positioned o er m awa from t e sensor in order for it to work


e s stem we are proposing in t is report would be bot low cost due to t e factit would be able to work wit an web cam wit out t e need for specialist ardwareand it will be capable of modeling an rigid structure Anot er difference from t eabo e s stems is t at t e camera itself can mo e around as well as t e ob ect we arecapturing ull motion capture is not in t e scope of t is pro ect

ssues

e problem is similar to a reconstruction from motion problem we will needto be able to determine t e skeleton model pose from t e information gat ered fromman iews around t e ob ect e must find a wa to accuratel do t is using as mucinformation as possible from t e iews t at are collected

apture and mapping of ob ect locations

e eral computer ision tec ni ues must be e plored in order to determine t e mostsuitable wa to find and accuratel place t e locations of t e markers w ic relate tospecific parts of t e ob ect e s stem will need to take into account partial occlusionof markers as well as ad erse lig ting conditions bot can a e a significant effect ont e accurac of t e pose estimations a ing a robust and accurate pose estimation oft e markers will elp reduce error furt er down in t e s stem and will t erefore resultin a better end pose

oise and camera lens distortion can also affect t e marker pose estimation e s stemmust be able to take t ese factors into account w en detecting t e markers and workingout t e pose of eac marker


stimating t e skeleton model pose

ince t e marker detection step will most likel not generate a accurate posesof t e markers and t erefore of t e parts of t e model we must a e a wa to allowfor t ese small discrepancies and opefull use information from ot er iews in orderto make t e final pose estimate as accurate as possible e optimal solution to t isproblem s ould be able to work out t e best pose estimate of t e skeleton model basedon bot t e marker locations and t e information about t e skeleton model pro idedb t e user

issing markers during t e marker capture stage must also be taken in account asedon t e structure of t e model definition it could be possible to get an accurate estimationof t e position of a part of t e ob ect e en wit out a ing seen or onl partiall seent e marker relating to t at part during capture

ontributions

is report outlines t e pat it took to get to t e final application w ic is a fullfeatured application t at can estimate t e pose of t e real world skeleton model inreal time using onl a web cam e will e plore t e initial approac es we tried beforegetting to t e final solution and e aluate w at worked and w at didn t work at eacstage e will t en go into dept about t e met ods and tec ni ues we used in t efinal implementation and also do an in dept anal sis about ow accurate t e s stemis inall we will do a guided walk t roug of t e s stem e plaining t e differentfeatures a ailable at eac stage e final application features a no el approac to t eproblem using bundle ad ustment tec ni ues to sol e t e problem of estimating t epose of t e real world ob ect t e results will s ow t e estimations are er accurateand can produce results w ic are onl a few millimeters off


ackground

n t is section we will take some time to e plain some of t e concepts and met odst at will be later used in t e implementation stage of t e report

omputer ision

omputer ision is t e field t at looks at e tracting information from images and orideo t picall from a camera of some kind is pro ect specificall deals wit using a

web cam to reconstruct t e skeleton pose of a real world model n a t pical computerision flow we start from low le el information suc as detecting of edges and regions

in an image and follow t roug to more ig le el reasoning about t e scene suc asow we can group toget er lines to form ob ects

n t is application we will be using a standard web cam in order to capture imagest ese images will be processed to find markers and we will make sense of t e ob ectin t e real world t roug t ese markers ince t e application is designed to be able towork wit an standard off t e s elf web cam t e c oice of web cam for t e pro ectwas not t at important elow we will go t roug some of t e t pical c aracteristicsused to differentiate between cameras:

ocal lengt in an optical s stem refers to t e measure of ow strongl t e lenscon erges lig t A s ort focal lengt bends lig t stronger and as a result allows t eoptical s stem to bring ob ects closer to it into focus n p otograp focal lengt ismeasured in millimeters e below image s ows t e effect of different focal lengt s:


ig : ocus lengt differences

or our s stem a large focal lengt would be optimal owe er we don t e pect t eweb cams t at will be t picall used for t e s stem will a e a large focus lengt mostoff t e s elf web cams a e uite a small focal lengt

e frame rate of a ideo camera s stem refers to t e rate of capture is is measuredin frames per second A ig is preferred in computer ision applications asa ig implies a fast s utter speed w ic results in less motion blur A commondownside of a ig is t at e posure is reduced posure is t e measure of owmuc lig t falls on t e image sensor in one s utter c cle a ing a low e posure meanst e image is less brig t and can result in t e image being undere posed t e image willa e loss of detail in darker areas

mage noise can be a problem in computer ision applications as it can result in loss ofdetail and or incorrect labeling in certain computer ision tec ni ues n t e conte tof digital ideo and p otograp noise is generall accumulated in t e image due top oto diode leakage in t e image sensor ow lig t conditions can cause greater noiseas t e camera s stem must compensate b increasing t e sensiti it of t e image sensorw ic can cause more noise due to leakage


amera model

t is essential in most computer ision s stems to be able to model ow t e camerapro ects t e real world iew onto its image plane e standard model t at is usedfor t is is t e pin ole camera model e pin ole model assumes t e real world iewis passed t roug a single aperture point before itting t e image plane be ind t eaperture point e relations ip between t e point [X,Y, Z] in t e real world andt e image plane in (u, v) coordinates is s own in matri form below:

λ

uv1

=

f 0 0 00 f 0 00 0 1 0

XYZ1

t can also be represented in t e following two e uations:

−u = fX

Z

−v = fY

Z

e focal lengt of t e camera is represented b f in t e formulas abo e e canalso con ert t e u and v alues to coordinates on t e image plane b taking t eproduct of u or v wit t e widt or eig t of t e image plane respecti el

e pin ole model is useful for er simple s stems owe er normall we wouldn trel on t e pin ole model b itself a ing a single point t at t e lig t passes t rougwould not allow enoug lig t for s ort e posures w ic is essential for ideo captures stems ameras use lenses to focus more lig t into t e aperture point to get aroundt is problem owe er t is introduces anot er problem n t e manufacture of lensesit is ine itable t at some sort of imperfection will occur due to t e near impossibilitof creating a perfect lens e will need to take into account a wa to correct t eseimperfections in t e camera model e first imperfection t at ma occur is t at t eoptical a is or center ma not matc up wit t e center of t e image plane t is canbe caused b improper fitting of t e lens bod to t e sensor and aperture ole ecan use t e following re ised formulas t at take t is into account:

x = fx(X

Z) + cx


y = fy(Y

Z) + cy

n t is form fx and fY are t e product of t e focal lengt and t e new parameterssxand sy w ic represent t e si e of t e sensor or image plane cx and cy representst e point w ere t e optical a is intersects t e image plane also known as t e principalpoint utting t e (x, y) coordinates into omogenous form allows us to represent t eabo e formulas in matri form as follows:

xyz

=

fx 0 cx0 y cy0 0 1

XYZ

is is known as t e intrinsic matri as it encompasses t e intrinsic parameters of t e

camera model ese parameters encompass focal lengt image format and principalpoint

e ne t step is to look at correcting for t e aberrations t at ma be present in t e lensitself we call t ese t e e trinsic parameters e two most effecting imperfectionsis t e radial distortion and t e tangential distortion e radial distortion is t e effectw ere t e furt er ou mo e awa from t e image center t e more stretc ed t e imageappears t is is t picall caused t e fis e e effect w ic can be seen in igure

adial distortion can be represented b an e panded a lor series t picall we onlconsider t e first few terms as t e radial distortion tends to be dominated b t e lowerorder terms in a t pical camera s stem n order to correct radial distortion we uset e formula below:

xu = xd(1 + k1r2 + k2r

4 + k3r6)

yu = yd(1 + k1r2 + k2r

4 + k3r6)

ere xu and yu represent t e corrected x, y coordinates xd and yd are t e distortedcoordinates ki is t e it radial coefficient and r =

√(xd − xc)2 + (yd − yc)2 w ere

xc, yc is t e principal point


ig : is e e


angential distortions occur w en t e lens is not fitted parallel to t e image planeit usuall appears as a sort of s earing in t e image e can remo e t is using t efollowing formula:

xu = xd + (2p1y + p2(r2 + 2x2))

yu = yd + (p1(r2 + 2y2) + 2p2x)

e ne t t ing to do is to work out t e all t e unknown intrinsic and e trinsicparameters A well known tec ni ue for doing t is was in ented b ang et alt in ol es taking man images of a multiple planar surfaces of known widt and eig tand optimising t e camera parameters until t e model we estimate matc es up witw ere t e planar surface points are in t e image an programs use is approac towork out t e parameters one of t ese programs is toolbo w ic we use for t isapplication

undle Ad ustment

undle ad ustment is a term used to describe t e problem of simultaneousl refiningt e location of coordinates seen in a number of iewpoints w ilst also refining t eparameters describing t e camera and t e structure of t e scene undle ad ustmentis almost alwa s t e last step of an feature based reconstruction algorit m Anumber of factors need to be taken into account w en deciding on t e strateg totackle t is problem or instance if we a e ero means aussian noise in eac imaget e problem becomes t e ma imum likeli ood estimator e problem essentiall boilsdown to working out to minimising t e re pro ection error of points in eac iewpointon linear least s uares optimisation is t picall t e c osen met od to tackle t is

problem

on linear optimisation

on linear optimisation is t e act of sol ing a set of e ualities s aring unknown parameters e aim is to tr and find t e parameters t at best fit all t ese e ualities weuse a ob ecti e function t at tells us ow well we are fitting t e e ualities enerallwe are looking to ma imise or minimise t e total cost of t e ob ecti e function eob ecti e function is also non linear in t e case of non linear optimisation e can


isualise t is as a n dimension grap w ere n is t e number of unknown parameterseac data alue represents t e cost of t e set of n parameters e lowest point in t egrap is t e solution as t is is t e set of parameters w ic corresponds to t e lowestcost w en put into t e ob ecti e function

A common approac to t is problem is to use e enberg ar uardt it is er common in computer ision applications t at need to do non linear optimisation e enberg

ar uardt aims to find a numerical solution to t e problem of minimising an ob ecti efunction o er t e space of a set of unknown parameters e problem it aims to sol ecan be described in t e below formula:

S(β) =

m∑i=1

(yi − f(xi, β))2

n t e abo e formula β represents t e set of unknown parameters we are tr ing tosol e (yi, xi) is t e pair of independent and dependent ariables and f represents t eob ecti e function

A is a iterati e algorit m to start it t e user must pro ide a set of guess parametersne problem wit A is t at its will terminate w en it reac es a local minimum due

to t e fact t at it can t differentiate between t e global minimum t at we want to findand a local minimum erefore t e c oice of starting guess parameters can be erimportant depending on w et er ou will be e pecting an local minimums or not ft ere are no local minimums t e c oice of starting guess parameters is not importantas t e optimisation will e entuall con erge to t e correct solution

e algorit m starts b appl ing a new estimated set of guess parameters and workingout t e new ob ecti e cost:

f(xi, β + δ) ≈ f(xi, β) + Jiδ

Ji =δf(xi, β)

δβ

f we reac t e case w ere t e cost function is minimised to ero we can see t at t egradient of S wit respect to δ will be ero as well


S(β + δ) ≈m∑i=1

(yi − f(xi, β)− Jiδ)2

f we now take t e deri ati e of t e abo e and set it e ual to ero we arri e at t efollowing formula in ector form:

( )δ = T [y − f(β)]

is t e acobian matri of j w ere t e it row of e uals Ji e can sol e t ee uation for δ e enberg also added anot er contribution t at elps speed up t ealgorit m e can use a damping function t at if t e reduction of S is slow we cantend towards using t e dentit matri as a product of δ instead of t e acobian isbrings in more in line wit t e gradient descent met od f t e reduction of S is rapidwe lower t e damping function w ic allows t e algorit m to perform more like t e

auss newton met od e damped ersion is s own below:

( + λ )δ = T [y − f(β)]

e problem wit t is is t at if t e damping function is er large t e in erting of t e+ λ becomes insignificant e final impro ement t erefore was to use a diagonal

matri consisting of t e diagonal elements of instead of is gi es us t e finale enberg ar uardt formula:

( + λdiag( T ))δ = T [y − f(β)]

A

A is a met od presented b eorge lein and a id urra for estimatingcamera pose in an unknown scene using onl a calibrated mono camera s stem tstands for parallel tracking and mapping and is called as suc because it processing ontwo parallel t reads one of w ic is performing real time tracking of t e scene w ilstt e ot er is building a map from t e information pro ided from t e tracking

e tracking part of t e met od assumes t at a point cloud of feature points asalread been created t processing a real time feed from a camera in order to maintainan estimate of t e camera pose in t e map A basic o er iew of t is is below:


ew frame from camera a prior pose is estimated from t e motion modelap points are pro ected onto t e image based on t is estimated poseof t e coarsest scale features are matc ed in t e imagee camera pose is updated from t e matc es

feature points are now pro ected onto t e image and searc edA final pose is obtained from t ese matc es

e tracking stage also detects A feature points f t e camera as mo ed a sufficientdistance and t e tracking ualit is deemed to be good a ke frame can be stored andpassed to t e mapping t read in order to impro e t e map

e mapping t read is concerned wit creating a point cloud of feature pointsso t at t e camera t read can re pro ect t ese points out and calculate t e estimatedcamera pose t does t is using bundle ad ustment tec ni ues to determine t e camerapose of all t e ke frames passed from t e tracking t read and poses of all t e mappoints seen in t ose ke frames e enberg ar uardt bundle ad ustment is used todo t is A problem occurs w en we are rapidl e ploring t e scene and man newke frames are being added A is a n3) problem w ic means t at an increase inke frames would cause t e time perform optimisation to increase rapidl e met oddoesn t e pect all ke frames to be s aring all feature points so it can use t is euristicin order to reduce t e number of ke frames to perform optimisation on per run of t eoptimisation is allows t e met od to concentrate on optimising newer ke framesas t e arri e An time t e camera is not e ploring t e mapping t read can use torefine older ke frames A limits t e local bundle ad ustment to ke framesw ic consists of t e newest ke frame and t e ke frames nearest to it

e final effect is an accurate camera pose tracking s stem for small A workspacest works er well if it can find a lot of feature points to track in t e scene as it asa lot of information to build an accurate map ne downside of A is t at mo ingob ects in t e scene can cause problems wit t e map as feature points ma be lost orincorrectl tracked


ig : A in action

arker detection

n computer ision it is common to look for certain elements in t e scene in order todetermine information about t e total scene ne wa of making a computer isionapplication more robust is to design t e elements t e application is supposed to lookfor to make t em as eas to track as possible and or to encode information aboutt e scene ese are called fiduciar markers and are used as a point of referenceor measure n t is application we will be using fiduciar markers in order to tell usabout t e pose of t e skeleton model e t erefore need to use a marker design t atwill allow us to determine t e pose as well as be able to distinguis between differentmarkers e also need to a e a s stem t at will be able to detect t ese markers fromt e web cam feed ere a e been man approac es to t is problem mainl in orderto perform augmented realit ne of t e first frameworks w ic sol es t e problemof marker detection and pose estimation is A oolkit

A oolkit

A oolkit is a augmented realit framework t at contains a full featured markerdetection process t was first released in and since t en t ere as been manen ancements and ot er A frameworks based on t e principles of A ool it emarkers t at A oolkit uses are black and w ite s mbols surrounded b a black s uare


wit a t ick w ite border An e ample of an A oolkit marker can be seen belowe s mbol inside is used to uni uel identif t e marker e use of t e black s uare

surrounded b t e w ite border makes it eas for t e detection process to find t emarker in t e scene

e marker detection process follows a number of steps res olding is first performedon t e image to e tract out t e black s uare of t e marker is is a simple lig tingbinarisation images wit an a eraged alue of less t an t e t res old becomeblack w ilst t ose abo e t e t res old become w ite

ontouring can t en be done to e tract out t e s ape of t e black s uare in t e imageine fitting is t en performed on t e contour to fit straig t lines onto it ese four

lines make up a uadrilateral from w ic corners can be e tracted

i en t e four corners we can estimate t e pose of t e s uare using t e co planarposit algorit m urt ermore we can recognise w ic marker we a e detected bperforming pattern recognition of t e image inside t e marker wit t ose stored in aknown dictionar

A oolkit as t e benefits of being relati el fast as its computer ision step is notcomputationall e pensi e and it as been ported to man different platforms owe erit lacks in some ke areas occlusion of t e marker is a problem as t e detection failsif e en some of t e marker is occluded due to t e fact t at occlusion will break t econtour fitting of t e black s uare Ad erse lig ting conditions are also a problem dueto t e use of t res olding to e tract out t e features eflecti e materials used tomake t e markers as well as low lig t conditions can cause t e t res olding step toincorrectl label certain areas of t e image

A A

e a e c osen to use A A marker detection w ic is a framework based on t eideas of A oolkit but wit some impro ements e ke impro ement is t e inclusionof t e edge detection met od of detecting t e outline of t e markers as outlined b

artin ir er

e process begins b di iding t e image into regions of pi els ac region isdi ided furt er b scan lines pi els across and down dge detection is performedacross t ese scan lines to determine a potential edge of a marker dges t at a e arenot black w ite are re ected we can c eck t is b looking at t e colour aluesof t e potential edge A A A approac is now used to group edges in a region


to form line segments wo random edges are c osen in t e region f t e twoedges matc in orientation and t is orientation matc es t e orientation of a line oiningt e two edges t e two edges form a pot etical line e ot er edges in t e regionare t en sampled to see w et er t e support t e line An edge supports t e line ifit matc es t e orientation of t e line and is close to t e line is is repeated mantimes in t e region and t e resulting lines are compared to see w ic ones a e t emost support e lines wit t e most support are c osen w ilst t e rest of t e linesare discarded e two furt est awa edges supporting t e line are c osen as t e endpoints t us forming a line segment e segments are t en merged if t e follow t esame orientation and t e pi els between t em are considered edge pi els matc ing t eorientation of t e segment using t e obel operator inall uadrilaterals are formedfrom oining up line segments t at a e end points close to eac ot er and a e a blackinside t e area formed b oining up t e line segments e corners of t e uadrilateralare represented b t e end points of t e line segments

erall t is approac is more robust t an t e t res old and contouring approac usedin suc frameworks are A oolkit t can andle ad erse lig t conditions better as it islooking for an edge instead of simpl rel ing on a single pi el alue for t res oldingurt ermore it can andle occlusion to some degree as onl corners need to be detected

to form a uadrilateral t e fourt corner can be estimated using t e incomplete linesegments forming off t e e isting corners

ig : dge marker detection


Anot er impro ement made was to use markers t at encoded t eir own in t epattern is was done using markers w ic a e a binar grid as t e pattern w icencodes to a number nstead of doing pattern recognition to find t e we cansimpl read t e off t e grid directl igure s ows an e ample of ow t e sare encoded

ig : arker s

otion capture file formats

ull motion capture o er man frames is not wit in t e scope for t is pro ect owe erwe plan to utilise t e file format t at motion capture s stems use to storeskeleton pose information as it allows t e output of our s stem to be easil portablebetween a lot of modeling tools e begin b e plaining some ke terminologt at will be used t roug out t is report in t e conte t of c aracter pose animation:

keleton

is refers to t e collection of all elements t at make up our c aracter

ones

A bone represents a basic entit in t e skeleton t is t e smallest element t at is sub ectto indi idual rotation and transformation e bones can be labeled for instance a bonein a skeleton can be called t e femur


egrees of freedom

ac oint in t e skeleton as a set range of mo ement t at it can mo e in or instancet e oint between t e upper and lower arm is limited to onl being able to mo e intwo directions

io ision ierarc ical data

consists of two parts e first describes t e ierarc of t e skeleton and t einitial pose e second describes t e mo ement of eac bone for e er frame ofanimation An e ample of t e first section of a file is below:

Ai p s

A p o s i t i o n p o s i t i o n p o s i t i o nr o t a t i o n r o t a t i o n r o t a t i o n

i g t p e g

−A r o t a t i o n r o t a t i o n r o t a t i o n

i g t ow e g

−A r o t a t i o n r o t a t i o n r o t a t i o n

i g t o o t

− −A r o t a t i o n r o t a t i o n r o t a t i o n

nd i t e

−

e abo e describes a skeleton wit t e root as t e ips e rest of t e bones aredescribed in a recursi e fas ion wit eac bones information and c ildren bones are


encapsulated in curl brackets n t e e ample abo e we a e a single upper leg boneappended onto t e ips w ose single c ild is t e lower leg w o as a foot as a c ildbone nd site refers to t e ending point of t e ierarc ote t at t e offsets arerepresenting t e oint location oining t e parent to t e c ild t is is s own in t epicture below

refers to t e relati e translation in and t at t e c ild bone as to itsparent n t e case of t e ips it refers to its translation globall ranslation refers tot e range of mo ement t at t e bone can a e relati e to its parent bone one lengtcan be inferred from t e offset information

e second section is s own below:

r ame s :r ame ime :

− − −− −

− − −

e abo e e ample s ows a single frame of motion e numbers on t e line belowframe time are t e rotations for t e bones e relate to t e c annel data pro ided int e first section or instance t e first numbers are for t e ips as it as degreesof freedom t e ne t t ree numbers are for t e upper leg and so on

is t e most widel used format for motion capture and is supported b mostma or modeling and animation tools t as some downfalls owe er ince bonelengt is not e plicitl defined t ere can be conflict if a bone as multiple c ildrenAlso t e file does not gi e details of t e en ironment suc as w ic direction pointsupwards


mplementation

is section deals wit ow t e s stem was implemented to ac ie e t e goals set oute will go o er t e initial approac es to t e problem ow t e solution e ol ed o er

time and finall describe in detail ow t e final solution was implemented and t ereasoning about t e c oices made o er pre ious approac es

ser efined odel

e first task was to define t e model t at t e user would be inputting into t e s stemA decision was made to base t is model on t e standard model e plained in t ebackground section is model was c osen as it was simple to understand and wasfle ible enoug to e press almost an real world rigid skeleton model e model fullcomplies wit t e standard and can be e ported to a file after obtaining afinal estimated pose

one model

A model will be defined from its root bone eac bone can a e an number ofc ildren bones ac bone will a e an associated uler rotation an offset fromits parent bone and a set of markers attac ed to t e bone t is easier to understandif ou t ink of a bone as t e bone oint positioned at t e offset from t e parent boneoint e root bone will alwa s be at position and a e rotation in t eand angles e transform of a bone oint in t e coordinate space wit t e centerat t e root and wit t e p ector represented as can be representedwit t e following formula:

n∏i=0

RiTi

ere n refers to t e current bone i is referring from t e root bone up to t e currentbone R is t e rotation matri and T is t e translation offset matri As suc t etransform is a affine transformation t at preser es t e lengt s offsets of all t e bones


ig : umanoid bone model

e global coordinate frame refers to t e coordinate s stem t at is centered atand is orientated wit t e p ector as t e oward ector as and t eig t ector as e offsets of eac bone refer to t e offset from t e

parent bone position in t e root bone coordinate frame

e skeleton pose refers to t e complete set of rotations on t e skeleton bone modele aim of t is application is to be able to detect and make an accurate estimation of

t e skeleton pose using t e real world model and a web cam

e entire bone structure is stored in bot a list structure as well as a tree structurew ere eac parent as a bi directional link to its c ildren t is also wort notingt at since we need calculate bone positions and marker positions uite regularl wea oid computing t e entire c ain of transforms e er time we want a specific bone bprecomputing and storing t e global transforms of parents before computing t e globaltransforms for t eir c ildren e list structure elps us easil ac ie e t is b alwa sstoring parents before t eir c ildren in t e list we can simpl iterate t roug t e listto compute global transforms


arkers

A total of markers wit uni ue s are a ailable to use t is is due to t e numberof uni ue rotational ariant patterns possible on a grid ese markers can beassociated wit a bone b marker offset from t e bone and t e uler rotationof t e marker in t e bones local coordinate s stem o find t e position of a markerin t e coordinate s stem of t e root bone is as follows:

(

n∏i=0

RiTi)MRnMTn

ere MR is t e marker rotation and MT is t e marker translation

A marker also as ordered corners o get t e ector location of a specificcorner t e final marker transform needs to be multiplied b t e offset of t e markercorner from t e center of t e marker e model defines t e edge lengt in mm oft e markers on t e model

ig : An e ample marker wit corners ig lig ted

roug out t is report we talk about t e marker coordinate frame t is refers to t ecoordinate s stem wit t e marker center as t e p and ig t ector beingplanar wit t e surface of t e marker t e and t e oward ector coming off t e top oft e marker

onstraints

e user can also define constraints on t e rotation or instance t e can limit t emo ement of a bone to onl rotate on t e relati e a is is is useful in situations


w ere t e real world model is actuall constrained to less t an degrees of freedom interms of rotation e rotation of t e knee oint is a good e ample it is constrainedto mo ement in onl a is as s own in igure

ig : nee oint

e default constraint is degrees on and e user can specif an w erebetween t ese alues for t e constraint e root bone is alwa s considered to beconstrained to all s in and for optimisation purposes t e rotation of t e rootbone does not particularl matter as we are onl concerned wit t e rotations relati eto t e root bone e benefit of constraining t e angles in t e model is t at t eestimation at t e end is likel to be faster and more accurate as it will not searc inregions wit angles t at are outside of t e constraining range

ile format

e application will output models into te t format w en t e user wants to sa e andload t e models up for future use e following igure s ows t e specification for t ebone model file:


[Name]&[ ParentBoneName ] *[ xLow ] [ xHigh ] [ yLow ] [ yHigh ] [ zLow ] [ zHigh ][ xro t ] [ yro t ] [ z ro t ][ o f f s e t x ] [ o f f s e t y ] [ o f f s e t z ][ ends i t ex ] * [ ends i t ey ] * [ ends i t e z ] *[ no of markers ][ markerID ] [ x o f f ] [ y o f f ] [ z o f f ] [ x ro t ] [ yro t ] [ z ro t ]

ig : pecification for bone model file

e asterisks indicate an optional alue in t e specification

e file is e ported wit t e figure e tension A simple e ample wit bones iss own below:

- - Model d e f i n i t i o n fo r : testmodel - -Hips-180 180 -180 180 -180 1800 0 00 0 010 0 0 0 0 0 0

2&Hips-180 180 -180 180 -180 1800 0 0-40 0 011 0 0 0 0 0 0

5&2-180 180 -180 180 -180 1800 0 00 40 012 0 0 0 0 0 0

ig : odel definition for testmodel

ideo input

irects ow is used for t e ideo input t e application accepts an standard web camt at is capable at running at frames per second and outputting at resolution


ac incoming frame is preprocessed to e tract out information suc as w ic markersare iewable and t e rele ant marker transforms A single frame and associatedinformation will be referred to as a iew t roug out t is report

ser nterface

A full featured was created in order for users to easil create load and sa et eir own models pon starting t e application t e user can c oose to edit an e istingmodel or create a new model e allows t e user to add edit or delete e istingbones associate dissociate or edit markers on a bone and see t eir c anges reflectedin real time wit t e interacti e model iewer pon finis ing editing or creatingt e model t e user can sa e t e model wit a c osen file name nteraction of t emodel iewer can be controlled solel wit t e mouse or wit t e mouse and ke board

e user can pan tilt rotate and oom in out using t e mouse urt ermore t epro ides an intuiti e wa to select focus on a particular bone using t e mouse

ig : ample of adding a new bone

one namenput for new bone offset and nameotation of selected bone


onnected bones of selected bonemodel iewer

ig : oading sa ed models

e t input bo odels s aring t e prefi wit t e name in t is bo areig lig tedodel iewer

e grap ics were implemented using A odel files a e t e e tension figure andare sa ed into separate folders wit t e name of t e model wit in t e director : ocuments figurate models odel ame odel ame figure sers can o erwritee isting models b specif ing an alread e isting model name w en sa ing t e will beprompted t at t e are o erwriting before t e confirm to sa e e also pro idesa wa to see t e ordering of corners on a marker impl select marker mode on t etop left corner and all t e markers on t e model will displa t e corner inde es

After model creation is done or an e isting model is loaded t e user can c oose to runone of two options e first is a test t at simulates running one of t e implementedrotation optimisation tec ni ues e plained in t e ne t section is can be run b


pressing t e button e second option is to perform t e optimisation using t eactual real world model and t e web cam e user will be able to see t e web camiew on t e rig t of t e screen and t e watc t e model being fitted to t e real

world model in real time on t e left of t e screen is can be started and stopped bpressing t e button n screen guides to elp t e user obtain t e best optimisationwill be displa ed depending on t e optimisation tec ni ue c osen

arker pose detection

All t e approac es re uired t e s stem to know w ere t e markers in t e ideo framewere relati e to camera e A A librar b was used as it pro ided met odsto detect markers in t e scene A A uses an edge detection approac in order todetect markers in t e scene A basic o er iew of t e process it uses to detect markersis as follows:

rab image from web camon ert to ra scalese adapti e t res old to binarise imageearc for edges in imageroup edges into linesind sets of four oined lines to get potential markererif marker b c ecking if outside is w ite and inside is blackransform uadrilateral into s uareetermine from pattern in s uare

A more detailed e planation is included in t e background section e four corners ofa marker can be found in t e scene using t is routine t ese corners can be ordered ina s stematic wa due to t e fact t at t e pattern inside t e marker is rotation ariantand t us can tell us t e rotational orientation of t e marker in t e scene rom t is wecan estimate t e camera transform relati e to t e marker position using t e coplanar

algorit m generates two appro imate poses in t e degenerate casew ere t e four input points are coplanar is routine is outlined below:


Algorit m

εi(0) = 0, n = 1eginning of loopol e for i, j and Z0 using t e below e uations en t e points are coplanar

t e additional e ualit i.j = 0 must be used wo poses are foundompute εi(n) =

1Z0

M0Mi · k, wit k = i× j n t e coplanar case two setsof εi wit opposite signs are foundf | εi(n) − εi(n−1) | res old itlse n = n+ 1 o to step

M refers to t e model points w ere M0 is t e reference point of t e ob ect and Mi

is t e position of t e it model point relati e to M0

M0Mi · I = xi(1 + εi)− x0,

M0Mi · I = yi(1 + εi)− y0,

wit

I =f

Z0i, J =

f

Z0j,

ig : ol ing for i and j

e second step is to discard one of t e appro imate poses is is ac ie ed b usingt e euristic t at all corner points must a e been in front in t e camera Zi > 0 tcan be seen t at at eac iteration two poses are obtained t is means we a e 2n posesafter n iterations n practice we can generall discard one of t e poses per iterationand t erefore onl a e to tra el done one branc of t e tree of possible poses Afterac ie ing t e t res old we obtain our final estimated pose

amera calibration

n order to get an sort of accurac from t e pose estimation of t e markers we musttake into account t e distortion of t e camera lens e model for camera distortionis e plained in more detail in t e background section A A can take into accountcamera distortion if t e user inputs an document t at tells it t e camera s intrinsicand e trinsic parameters toolbo was run wit two marker boards of si ebo si e mm and images in order to calibrate t e camera e results from

toolbo were t en inputted into t e file for use wit A A


ig : camera calibration


nitial approac es

n t is section we will e plore some of t e initial approac es tried before coming upwit t e final solution e ad antages and disad antages of eac approac will be listedand t e features carried forward into t e final solution will also be ig lig ted acof t e approac es was based on t e idea of sol ing bundle ad ustment using non linearprogramming t at is to sol e a set of e uations collecti el o er a set of unknownariables wit some sort of ob ecti e function t at must be minimised is ob ecti e

function is non linear e tec ni ue used to sol e t ese e uations and minimiset e ob ecti e function was c osen to be t e e enberg ar uardt tec ni ue w ic ise plained in dept in t e background section of t is report e reason w we c oseto use non linear programming was because we want to take as muc information fromt e scene into account ac iew from t e web cam can onl see certain parts of t emodel and we wanted a wa to collecti el use different iews in order to estimate t erotations of eac bone urt ermore t e final transform of eac bone is dependent ont e transforms of its c ain of parent bones all t e wa to t e root As suc we knowt at eac e uation will take into account a set of bone rotations e can t erefore knowt at t e unknown parameters we will be optimising o er will be t e rotations foreac bone o er t e range of degrees

A approac

e first approac in ol ed using A in con unction wit marker detection in orderto generate a map of w ere t e markers were in t e unknown scene A would beused in order to estimate t e global camera pose w ilst e ploring t e en ironment witt e web cam e camera model used is t e modified pin ole camera model describedin t e background section is approac is outlined below:

or e er iew obtained from t e web cam markers are detected and eaccorner is stored as a pi el coordinate as to w ere it appeared on t e iew

is is stored alongside t e camera parameters t at describe t e camera posein t e global coordinate frame e [X,Y, Z] look at ector and t e [X,Y, Z]camera positionf we see t e root bone we know we are seeing t is if we see t e markerassociated wit it we can t en work out t e transform to get from t e globalcamera coordinate frame to t e root bone coordinate frame e t en know fore er frame w ere t e camera is in relation to t e root bone position e candetermine from t is w ere all t e ot er marker corners s ould be on t e screenbased on t e estimated rotation alues


e ob ecti e function aims to minimise t e sum of t e euclidean distance betweent e pro ected pi el coordinate of a specified corner from t e estimated modeland t e same corner as seen in t e iew of t e web cam across all selectediews iews are randoml selected from t e set of all iews for eac run of

t e optimisation e ob ecti e function is summarised in igure

m−1∑j=0

(3∑

i=0

(MEij −MVij)2)

ig : b ecti e function to minimise

ere m is t e iew number i is t e marker corner inde MEij is t e pro ectionof t e it corner in t e estimated skeleton model onto t e screen using t e camerapose in t e jt iew MVij is pi el coordinate on t e screen of t e it corner in t ejt iew

e can find t e pro ection of a specified corner b first finding t e translation ectorin t e root bone coordinate frame from t e estimated model refer to section andusing t e e uation in igure

u =Xf

Z, v =

Y f

Z

x =1

2Width+ u

1

2Width , y =

1

2Height+ v

1

2Height

ig : ro ecting corner onto iewing plane

ere (u, v) are coordinates on t e iewing plane (x, y) is t e pi el coordinates ont e screen f is t e focal lengt in mm and (X,Y, Z) is t e coordinate of t ecorner in t e frame of t e camera space Width and Height refer to t e widt andeig t of t e screen in pi els

ac run of t e e enberg ar uardt optimisation takes iews t erefore t e total

number of e uations is e ual to 30∑b

i=0 8Mb w ere b is t e bone count and Mb

is t e number of markers on bone b ere are e uations per marker as we taket e and coordinates for eac corner as separate e uations to optimise o er wasc osen as t e iew count as t e e enberg ar uardt algorit m is of t e order O(n3)

was a good compromise between number of iews and computation time for t ea erage model ere were also problems related to local minimums if we selected too


man iews e number of iews taken can be tweaked in t e est e enberg A csclass file if t e model is er large or er small

mplementation e first step taken was to compile and run A on indows t iswas done in isual tudio After a build of A was ac ie ed work was doneto make A as a separate process be able to communicate wit t e main applicationrunning in amed pipes was c osen as t e met od of communication as it ad alow o er ead and was ideal because of t e fact t at bot processes would be runningon t e same mac ine A was modified wit an e tra t read t at acts as a ser erfor t e named pipe communication additionall an e tra procedure was added to t emain tracking loop w ic detects markers in t e scene and estimates t e pose of t emarkers ince A alread grabs frames from t e web cam it was not needed fort e application to grab camera frames of its own

pon connection to t e ser er as a client t e main application would start re uestingiews from A A would be passing to t e client a image

representing t e current frame as well as t e associated camera pose and also t emarker transforms and s in t e current frame is would t en be stored in t emain application as a new iew for possible use in a future optimisation e cameraiew from A can be acti ated in t e main application b clicking on t e Acti ateA button in t e bottom left of t e en t e user specifies to start t e

optimisation an incoming frames are sa ed as iews if t e camera pose stabilit isdeemed good b A en t e user presses again e enberg ar uardt isused to estimate t e rotations of t e bones using of t e iews selected at randomfrom all iews e optimisation can be repeated wit anot er set of iews bpressing again e old estimated rotations are used as t e starting guesses for t enew run of e enberg ar uardt e camera images are displa ed to t e user inreal time

A simulated test was also created for t is approac t started b rotating t e bones atrandom t is would be t e result we are wanting to get to at t e end of optimisatione t it generated random camera poses around t e skeleton model or eac iew

t e skeleton model wit t e rotations applied was pro ected onto t e screen using t ecamera pose associated wit t at iew aussian noise was t en applied to t epositions of t e corners in eac generated iew ese generated iews were used asinput to t e e enberg ar uardt simulated run e reason for creating t is simulatedtest was to determine w et er t is approac would be iable from an earl stage andto easil determine ow accurate we could get wit t is approac e could easildetermine t e rotational error for eac bone and t e point difference for eacmarker corner between t e estimated and actual model


aluation rom t e simulated runs we uickl determined a problem area for t isapproac rom running t e simulation times about of t ese runs resulted in erlarge rotational errors w ilst t e ot er resulted in acceptabl close radiansrotations per bone e cause of t e bad runs was due to t e fact t at e enberg

ar uardt was stuck in a local minimum w en optimising o er t e set of all e uationscausing t e algorit m to terminate wit a bad estimation A number of possible fi eswere outlined to sol e t is issue:

andom starting parameters e would be running simultaneous instances ofe enberg ar uardt wit randoml c osen starting parameters and selecting t e

instance wit t e minimum ob ecti e function after all t e instances return edownside to suc an approac would be t e computational o er ead of runningmultiple optimisations at oncese marker rotations to determine starting parameters e orientation of one

marker to anot er marker on t e parent bone allows us to roug l determine t erotation of t e parent bone e can use t is as input into t e optimisation

oose iews t at will produce better results ocal minimums can be reac eddue to poor selection of iews f we ne er see an of t e markers associatedwit a bone and t e bone as bot a c ild and parent t ere can be multiplepossible rotations possible resulting in man local minimums e can use someeuristic to c oose good iews to use for a run of t e optimisation

ome of t ese impro ements would be implemented in future approac es

e ma or problem associated wit t is approac was disco ered w en we attemptedto gat er t e iews from A e tracking information obtained from A wasnot er stable resulting in bad estimation of t e global camera pose and as a resulta bad pro ection of t e marker corners Anot er problem was t at A wouldlose t e global map uite often e er time t is appened t e root marker would a eneeded to be seen in t e iew of t e camera in order to use t e iews captured witt e new map ere were a number of reasons w t is was appening:

ot enoug feature points in scene A works best w en t ere are a lot offeature points to track across frames t e best feature points come from te turedand cluttered en ironments ince t e application would be t picall used w ent e web cam is looking at a mostl w ite ob ect on an empt desk it waslikel a ing difficulties getting a large number of good feature points to trackAdditionall t e camera would t picall be closeup to t e ob ect and rotatingaround it e feature points cannot be tracked w en t e ob ect is occludingt em w ic would appen a lot as t e camera mo es in a pat around t e ob ect


As a result of a ing onl a small set of alid feature points tracking ualit wasdegradedide angle lens A works best wit a wide angle lens as it can see more

of t e en ironment in a single frame and t erefore find a larger number of alidfeature points to track e application is made to work wit onl a standardweb cam wit a normal t is issue compounded wit t e abo e issue furt erdegraded t e tracking ualitel ing on t e root bone marker iews in a map t at asn t seen an of t e

markers attac ed to t e root could not be used in t e optimisation step ecamera pose used in t e optimisation was t e global camera pose transformed intot e root bone coordinate frame meaning t at t e ualit of a iew relied ea ilon t e map tracking t e pose of t e root bone accuratel t would be possibleto make t e optimisation root bone pose in ariant b not specif ing t at t e rootbone was at position wit no rotations and allowing optimisation to runwit iews using t e global coordinate frame owe er t is introduced moreproblems as it caused more local minimums due to t e fact t at t e skeletonroot bone was not fi ed in position

ome of t ese problems could a e been remedied b a ing t e user trained in owto mo e t e camera as well as modif ing t e en ironment to make it easier to get goodfeature points suc as placing a c eckered pattern underneat t e model but it wasdecided t at t is would be against t e principle of t e application being able to workalmost an w ere and b an one wit out knowing an tec nical knowledge about it

is approac was ultimatel abandoned due to t e after mentioned problems alt ougsome of t e ideas discussed abo e would appear in later approac es

arker distance approac

e ne t step was to look for a ob ecti e function t at did not rel on global cameraposition e upside of t is would be t at we would not need to repl on an sort ofaccurate tracking of t e global camera pose t e downside was t at we would need torel on a ing or more markers in t e frame at a time ere would be no wa ofconferring information about a iew wit onl a single marker due to t e fact we don tknow ow t e camera is orientated in relation to t e ot er markers on t e model

e intuition be ind t is approac was t at t e distances in between t e markerswould gi e us some idea about t e rotation of t e bone in t e model ased on t isan initial ob ecti e function was de ised:


m−1∑i=0

n−1∑j=0

(| Ti − Tj | − | Di −Dj |)2 i ̸= j

ig : istance cost function

ere Ti refers to t e ector location of t e center of marker i detected in t ereal world iew in t e camera coordinate frame Di refers to t e ector locationof t e center of t e marker i from t e estimated model in t e root bone coordinateframe and m,n represent t e marker count

is new cost function was inputted into t e simulation and test data was generatedfor a simple bone model in order to test t e iabilit of t e approac pon runningt e tests it became clear t at t e cost function ad no wa of determining t e localrotation of a marker since it was onl rel ing on t e distances between t e markercenters is problem was most ob ious on t e end sites of t e skeleton model w ereif t e marker was positioned at t e bone oint t e bone could an infinite number ofpossible rotations t at would lead to t e same cost

e ne t impro ement was t erefore to not rel on t e distances between t e centerof t e markers but to look at t e distances between associated corners on markersAssociated corners are corners t at a e t e same corner inde on different markers

is allows t e optimisation to take into account local rotation information t at maa e been lost b onl looking at t e distances between t e centers of t e markerse ob ecti e function is s own below:

m−1∑i=0

n−1∑j=0

3∑c=0

(| Tic − Tjc | − | Dic −Djc |)2 i ̸= j

ig : orner distance cost function

e ariable c refers to t e corner inde

mplementation

nstead of looking at indi idual iews and minimising t e distance cost for eac iewit was decided to minimise t e a erage distance of all iews for a pair of marker corners is was done to reduce computational cost instead of optimising V iewcount ·∑m−1

i=0

∑nj=0 4MN e uations we would be optimising o er

∑m−1i=0

∑n−1j=0 4MN

e uations e application keeps a running a erage of distance for e er corner association of e er marker to marker relations ip t is information is stored in t e


istance torage cs class file eac corner for a marker to marker relations ip as a bint at olds its running a erage en a new frame is captured t e markers are detectedt e distances are calculated and t en t ese distances are added to t e running a erage

e rotation of t e root bone was constrained to all s as t e orientation of t e rootdid not matter for t is approac likewise t e rotations for t e bones were constrainedto −180 6 r 6 180 degrees e flow of t e tracking part of t is approac is s ownbelow:

apture frame from web camdentif markers in frame compute transforms for markersork out corner distances for markers seen add new distances to t e rele ant

running a erage of distances

imilar to before t e new cost function was simulated wit test data est data wasgenerated b taking t e actual distances from an alread rotated skeleton model andappl ing aussian noise to t ese distances ince a pair of markers ma ne er appearin t e same iew we added in a probabilit t at a pair of markers ne er a e anrecorded distance into t e test data as well

or t e real world running we implemented anot er processing t read t at continuouslran t e optimisation in t e background w ilst updating t e model in t e modeliewer wit t e new estimated rotation alues is allowed t e user to see ow

close t e optimisation is getting to t e real world skeleton model in real time isoptimisation t read is summarised below:

op t e arra of a erage distances for all marker marker relations ips t esewill be used as our obser ed data for t e optimisation e uations

et pre ious estimated rotation parameters nitial run will be set to all sun optimisation for iterationspdate model in iewer wit new estimated rotationso to

aluation

is approac produced ade uate results er uickl but t ere were problems witgenerating er accurate results especiall at t e e tremities of t e skeleton model norder to test t e real world running of t is approac a couple of test skeleton modelswere created known rotations were applied to t e model and t e application was runagainst t e rotated model A record of a erage rotational difference per a is on a bonewas recorded against t e number of iterations of t e optimisation igure s ows t etest model and t e rotated test model igure s ows t e results of a run wit o er


iterations w ic took about seconds to complete otice ow t e rotationse entuall con erge down to about degrees a erage difference per rotation ebiggest rotational differences are located at t e e tremities of t e skeleton model asseen in igure e camera was constantl rotating around t e model

ig : eft nputted model ig t odel wit rotations applied

ig : esults of t e distance corner optimisation


ig : owing t e final result of optimisation


e results are promising for t is approac owe er t ere were some ke issues in ol edwit onl looking at t e distances:

erestimation of distance istances recorded between corners tended to beo erestimated e amount of error was dependent on t e distance of t ecamera to t e markers and also t e angle of t e camera relati e to t e markers

erestimation of t e distance caused problems for t e optimisation and is t ereason w t e end sites of t e skeleton model were t e most error prone twas unlikel t at an end site would a e anot er marker on t e ot er side ofit to compensate for t e o erestimation from one side of t e model t is causedt e optimisation to c ange t e rotation incorrectl in order to accommodate t eo erestimation of distance is problem would be e ibited b an markerw ic is located in an area of t e model w ere onl a few ot er markers can beseen and t ese ot er markers are all located close toget er e o erestimation ofdistance is likel caused b not perfect calibration of t e camera in combinationwit t e fact t at t e marker detector cannot perform sub pi el accurac w endetecting t e marker causing t e markers to be detected slig tl too small iserror would increase as t e marker appearing in t e frame got smallerocal minimums ere were still cases w en t e optimisation got caug t in

local minimas t e optimisation works best if all markers can see eac ot erf a certain set of markers cannot see anot er set t ere ma be man possiblerotations t at would be alid in terms of reducing t e ob ecti e function

e upside of t is approac was t e speed of running t e problem si e was essentiallfi ed as instead of rel ing on separate iews we relied on t e a erage distances foreac corner of a marker marker relations ip e downside was t at it re uired ormore markers on t e screen in order to use t at particular iew Anot er downsidewas t e fact t at t ere were still uite a few rotational errors in t e final results mostldue to t e o erestimation of distance t ese rotational errors increased dramaticall ift e optimisation did not a e distances recorded for certain marker marker relations

inal approac : arker pro ection from iew

A final approac was de ised t at took inspiration from bot initial approac es estill did not want to be reliant on t e global camera pose owe er we wanted to beable to take eac indi idual iew into account and be able to optimise o er a numberof selected iews nstead of looking at t e distances between corners in eac iewwe pro ect out t e marker corner positions from a selected marker in t e scene andaim to minimise t e euclidean distances between t e seen marker corners and t epro ected marker corners igure s ows t e inputted model s own in igure


ig : ample s owing t e pro ected marker corners based around t e marker wit

pro ected onto a real world model wit out an rotations applied e pro ection isbased around t e marker wit orange points are t e pro ected corners red pointsare t e detected corners

b ecti e function

e first needed to define t e new ob ecti e function to minimise e ob ecti e functionwill be looking to minimise t e distances between w ere t e model e pects markercorners to be from t e estimated rotations and w ere t e are actual seen in t e framet is is a similar approac to t e A approac after mentioned e ke differenceis t at instead of using a global camera pose we will use a camera pose w ere asingle marker is selected to be at t e center e estimated skeleton model will betransformed so t at t e same marker on t e skeleton model will be matc ed up witt e marker seen in t e camera f we know t e transform matri of a marker in t eiew we can transform e er marker corner in t e skeleton model as in igure to

obtain its coordinate in t e c osen markers coordinate frame


T = M(C−1)WP

V = [T40, T41, T42]

ig : etting ector position of a marker corner

ere T is t e transform matri of t e marker corner in t e camera coordinate frameand V is t e ector coordinate of t at marker corner M is t e marker cornertransform in t e root bone coordinate frame C is t e marker transform of t e c osenmarker in t e root bone coordinate frame W is t e world transform matri i e t ecamera transform obtained from t e c osen marker and P is t e pro ection matri

nowing t e ector position of t e marker corner in t e iew we can now do apro ection to get it into screen coordinates using t e formula in igure

u =Xf

Z, v =

Y f

Z

x =1

2Width+ u

1

2Width , y =

1

2Height+ v

1

2Height

ig : ro ecting to screen coordinates

ere (u, v) are coordinates on t e iewing plane (x, y) is t e pi el coordinates ont e screen f is t e focal lengt in mm and (X,Y, Z) are t e coordinate of t ecorner in t e frame of t e camera space Width and Height refer to t e widt andeig t of t e screen in pi els

e actual marker corners seen in t e iew can be e tracted from t e rele ant markertransform matri ince we know t e edge si e of a marker we can take t e corner asa translation from [0, 0, 0] and multipl t is translation matri b t e camera transformfrom a specified marker e can t en use t e same pro ection formula as in igureto get t e position in screen coordinates

ow t at we found t e two marker corner screen coordinates t e ob ecti e functionis to minimise t e distances between t em is lea es us wit two problems firstlow do we c oose w ic marker is to be used as t e c osen marker in eac iewecondl ow do we c oose w ic iews to use

e first problem was sol ed b ranking t e markers in eac iew depending on owaccurate its pose estimation was likel to be e intuition was t at since some of


t e marker corners were far far awa from t e c osen marker position an error inpose estimation would be e acerbated t e furt er t e point was awa from t e c osenmarker oosing t e most accurate marker meant t at we reduced t e likeliness of aer bad pose being used w ic would result in a lot of error w en calculating t e

point difference si e of t e marker in t e iew was used to rank ow accurate t epose estimation for a marker was likel to be e larger t e area on t e screen t eless likel t at sub pi el inaccuracies w en detecting corners would occur igures ows t e formula to get t e area of t e marker in t e iew

A(Θ) =1

2| (V2 − V0)× (V3 − V1) |

ig : Area of a uadrilateral

ere Vi is t e coordinate of t e it point

e can define t e ob ecti e function in terms of t e set of e uations for eac iewe use t e abo e ranking to obtain t e marker t at is most accurate wit in a iew

t e ne t step is to pro ect t e estimated skeleton model based on t e c osen markere ne t step is to pro ect out t e corners for eac seen marker in t e iew inall

we can get t e distance as s own b t e formula in igure

n−1∑i=0

3∑c=0

(Eic − Vic)2

ig : distance between marker corners

ere n refers to t e marker count c is t e corner inde Eic is t e calculated screencoordinates of marker i and corner c from t e estimated skeleton model and Vic is t ecalculate screen coordinates of t e marker i and t e corner c in t e iew

ince t e computational time for t e optimisation increases e ponentiall wit t eproblem si e as it is an (n3) problem it was decided to limit t e iews per runof t e optimisation to iews Anot er impro ement is to not alwa s run a globaloptimisation looking to optimise t e rotations for all t e bones in t e model e canrun localised optimisations t at onl looks at a subset of bones to optimise o er eskeleton model makes it eas to di ide up t e optimisation into localised sections sincewe know t at to estimate t e rotation of a bone onl t e bones in t e route to t eroot bone would affect its final position is is illustrated in igure


ig : ocal optimisation

otice ow a rotation of t e parent bone of t e top alf will not affect an of t e positions of bones in

t e bottom alf

f we represent t e skeleton model as a tree we can perform a local optimisation bmo ing from t e rootbone and picking our wa down t e tree until we reac a leafnode A leaf node represents an end site of t e skeleton model is pat t roug t etree can be furt er reduced in si e b picking a connected section wit in it to performt e optimisation o er e would onl need to consider markers t at are related to t ebones in t is selection e iews t at we would use for t e local optimisation musta e or more of t ese markers in t em ot erwise t e would be wort less to t e

local optimisation referabl we want t e iews w ere t e most accurate marker int e iew is one of t e markers in t e selection in order for t e results from t e iewto be less error prone

An outline of t e localised selection is below:

tart at root nodeelect one of t e c ildren add to selection

of adding an of t e ot er c ildren to selectionra el to first selected c ildf c ild is end site terminate else go to


ince t e computation time of t e optimisation is e ponential wit t e number of bonesb doing a local optimisation o er a subset of t e bones we are drasticall reducing t etime it takes per iteration of t e A optimisation

ig : one tree representation

nl t e ig lig ted bones will be affected b t e rotations of t e c osen bone

ere is one more euristic we can take into account to minimise local minimumproblems wit t e optimisation step e can take ad antage of t e fact t at we canestimate t e orientation difference between t e two connected bones b comparing t ecamera transform matrices between t e markers on bot bones e can use t is alueto constrain t e searc for t e rotation to up to degrees from t e estimatedalue e onl want to do t is if we a e ade uate data to support t at t e rotation

estimation is correct so we take t e a erage of all iews s owing t e rotation and onluse it as a constraint if we a e o er iews adding up to t e a erage e alsokeep a running ariance alue for eac rotation based on t is ariance we can limitt e constraint e en furt er a low ariance can make t e constraint as low asdegrees for t e rotation


(MiR−1i )(MjR

−1j )−1

ig : inding rotation transform from one marker to anot er

ere Mi refers to t e marker transform of i and Ri refers to t e rotation matri part of

t e transform of i

e e uation count for optimising across all bone rotations is t erefore V iewcount∑n

i=0

∑4c=0(Eic−

Vic)2 ereas a local optimisation is t e same e cept t at n represents onl t ose

markers included in t e local optimisation

mplementation

e can now outline t e entire algorit m in two sections e first section dealswit t e tracking part w ere we detect t e markers in a iew and process t e iewin preparation to be used in optimisation e second section deals wit ow t eoptimisation c ooses t e iews it is going to use and ow it determines t e constraintsto use for eac rotation parameter

racking

ene er a new frame is captured t e following appens:

etect markers in scene e use A A marker detection in order to detectmarkers and associate an and camera transform to t em

eck distances e can know t e ma imum possible distance two markerscan be apart from t e skeleton bone model is is essential t e ma distancet at t e markers can be apart in t at t e skeleton model allows e canprecompute t is distance for eac marker marker relation before t e optimisationbegins f a distance between two markers is larger t an t is ma imum distanceb more t an mm we can discard t e entire iew as it is likel t at t e markerdetector as incorrectl labeled one of t e marker s f we know t at atleast one marker as been incorrectl labeled it is likel t at t e tracking oft e iew was not ideal an wa s imilarl we can discard an iew w ere t eclosest marker appears more t an mm from t e camera t e pose estimationis unlikel to be er accurateork out rotation between connected bones f two markers on connected bones

appear in t e frame we add t e rotation between t em to t e runninga erage of rotation for t at particular bone pair in t e iew torage cs class file

e also increment t e count seen for t at bone pair rotation


ank marker accurac e rank markers wit in t e iew b t e area t at t euadrilateral formed b its corners in make it is more likel for a marker

pose to be accurate if it appears larger in t e iewonstruct iew ob ect A new iew ob ect is created t at includes information

suc as t e markers seen in t e iew and t e associated camera transforms isiew ob ect is stored along wit all ot er iews in t e iew torage cs class filee iew is placed into a bin t at corresponds to its most accurate marker A

reference of t is iew is also placed into a bin corresponding to its nd mostaccurate marker

ptimisation

ptimisation appens in a separate t read er run of t e optimisation doesiterations wit iews

ecide w et er to do local or global optimisation ere is a c ance ofdoing eit er f we c oose to do a local optimisation we use t e selection processin order to select t e bones to optimise o er rom t is we can t en modif t ee uations to onl take into account markers on t e c osen bone and t e bonesin its sub treeetermine constraints ere are two constraints t at can be applied irstl we

can look at t e user inputted constraints secondl we can look at t e constraintsimplied b t e a erage rotations seen in t e tracking stage f t e two constraintsare not o erlapping we take t e user inputted constraint as t e constraint foroptimisation and we also clear t e rotational a erage determined from t e trackingstage we assume t at t e user as constructed t e model correctl and t at t eseen rotation is impossible on t e model t erwise t e final constraint is t elower and upper bound from t e rotational and user inputted constraints t at ast e least differenceollate iews e select random marker s and pick iews t at a e t ese

marker s as t eir primar or secondar accurate markers we bias t e selectionof iews to pick ones t at a e been seen more recentl t is is to a oid nottaking in new information once t e iew count gets too large is gi es usa good spread of iews as t e logic to pick t e iews is not skewed towardsmarkers t at are seen a lot f we are doing local optimisation we can onlrandoml pick marker s corresponding to t e markers of t e bones we areoptimising Also we don t pick iews t at do not a e or more of t e markerswe are optimising o erun optimisation e can now perform t e optimisation e input estimated

rotation parameters are t e results from t e pre ious run all s for initial run


or e er iew we take two sets of e uations t e first set uses t e most accuratemarker as t e c osen marker t e second set uses t e second most accurate markerf we are working out t e ob ecti e function on a iew t at does not containa marker t at we want to include in t e optimisation t e difference for ane uation t at includes t at marker is set topdate model inall we can update t e model iewer wit our new

estimated rotation parameters

Anot er impro ement o er pre ious approac es is t at we can now see clearl w ent e estimation is poor in t e t now o erla s onto t e camera feed t e estimatedmarker corner positions centered on t e most accurate marker in t e current iew asseen in igure e points are re distorted using t e brown s distortion modelso t e matc up wit t e marker corners on t e screen

ig : s owing estimated corner positions


aluation

e e aluation of t e final application will be done wit uantitati e anal sis as well asualitati e anal sis e will be looking at ow accurate t e estimation of t e rotations

gets on a number of skeleton models w ic we know t e e act rotations of e willalso look at eac of t e impro ements to t e core optimisation e plained in t e pre ioussection and see ow t e affect t e final estimation result inall we will be doinga walk t roug of t e final application on a more complicated model to s ow owintuiti e it is to use

uantitati e anal sis

e will be performing t e optimisation on eac model for runs of iterations eace attac ed grap s will s ow ow t e a erage rotational difference for a single rotation

c anges o er t e runs e algorit m does not e pect t e same iews for e er runand t ere are non deterministic parts of t e algorit m as well so we do not e pecte er run to generate t e same results o elp alle iate differences due to c angingobser ations and randomness we will be doing passes of runs for eac model andtaking t e a erage of t e results of all passes e camera will tr and keep on a similarpat for e er pass performed in order to gi e t e optimisation similar iews for e erpass owe er t ere is still some randomisation due to noise and iew selection ecamera will be mo ing in a circular pat abo e t e model w ilst directed towards t emodel as s own in igure e will be using two main metrics for t e e aluationfirstl we will look at t e point difference is is done b looking at t e a eragedistance between w ere t e bone is estimated to be in t e skeleton model and w eret e bone actuall b appl ing t e pre measured rotation alues to t e skeleton modelwe take a reading of t is e er run econdl we look at t e a erage rotational errorfor an a is of rotation on a bone t is is t e difference between t e estimated rotationand t e actual rotation of t e real world model t is is recorded e er run as well ewill also look at t e ariance against runs for bot metrics


ig : amera pat

ase model markers

e will start b testing on a model wit markers igure s ows t is model witno rotations applied o constraints a e been applied to t e bones since we want totest t e accurac on all a is and

one nde offset offset offset constraint constraint constraint

A A AA A AA A AA A AA A AA A AA A AA A AA A AA A A


ig : marker model

e will now test wit t e rotations as s own in igure

one nde


ig : marker model rotations


ig : After optimisation

e first grap s ows t e a erage point difference for eac bone in millimetersotice t e rapid con ergence after less t an iterations is is mostl due to t e

rotation constraints t at are applied to t e optimisation t is also wort noting t at t eariance of point difference for eac run was progressi el getting low eac bone

was e ibiting roug l t e same point difference due to t e fact we are optimisingo er all markers in a iew f we onl took into account rotational information acrossconnected bones we would e pect a larger point difference near t e e tremitiesof t e model At t e end of t e optimisation we can see t at bones came to roug lmm of t e e pected position w ic is an e cellent result e mm error could be

due to a number of factors suc as s stemic error due to calibration or noise


ig : point difference

ig : point difference ariance


e grap in igure s ows t e a erage rotational error for eac rotational a is oneac bone o er t e number of runs performed t roug l follows t e same s apeas t e point difference grap wit a rapid con ergence towards t e correct posearound t e t run t iteration and t e stabilisation towards t e t run t alsos ows we can e pect t e optimisation to reac a state w ere eac rotation is roug l

degrees off t e e pected rotation nlike t e point difference we noticed agreater ariance on rotational errors t is is likel due to t e self correcting nature oft e optimisation e point differences are reduced to obtain a more acceptablegeneral pose sometimes at t e cost of worse rotational error on some of t e bones int e model

ig : otational error


ig : otational error ariance

e second test will use t e same model wit different rotations applied as in igureis test was also performed under less t an ideal lig ting conditions

one inde


ig : marker model rotations

ig : After optimisation


igure s ows t e point difference t followed a similar s ape to t e pointdifference from before owe er t e con ergence is not as pronounced otabl t isoptimisation was done in less t an ideal lig ting conditions w ic would a e causedproblems wit t e ualit of a ailable iews oise would a e pla ed a bigger factorcausing t e marker pose estimations to be of poorer ualit Anot er problem wouldbe t at some markers ma a e gotten mistaken for ot ers is is reflected b t e lessstable optimisation s own in t e grap a ing a good selection of iews would a e amuc greater importance t an before due to t e fact t at some iews could ad erselaffect t e final result of t e optimisation

ig : point difference


ig : otational error

e will now look at ow t e rotation constraints affects t e optimisation stage of t eprocess e ne t test will be using t e same rotations as in t e first marker test

owe er we will be turning off t e rotation constraints and allowing t e optimisationto use an alue between and degrees w en estimation t e rotations


ig : inal estimation wit no rotation constraint

igure s ows t at e en t oug t e rotation constraints were remo ed we a e stillobtained an accurate estimation of t e skeleton pose t ma seem t at t e rotationconstraints ma not add an t ing to t e process owe er it does impro e on owfast t e optimisation con erges to an accurate and stable solution otice ow t eoptimisation took nearl runs to become stable wit an accurate result compared tot e runs needed for t e optimisation wit constraints applied is is because t eoptimisation as to searc across a larger range to get to t e final estimated rotation

sing t e rotation constraints can also reduce t e likeli ood of local minimums affectingt e final pose due to t e fact t at t e optimisation will start near to t e actual skeletonpose instead of starting at all s for rotations igure s ows t e rotational error itis er ig initiall due to t e fact t at we are starting at all s rotation and e entuallgets close to t e accurate rotation alues


ig : point difference o rotation constraints

ig : otational error o rotation constraints


ualitati e anal sis

n t is subsection we will walk t roug t e process of using t is tool from end to ende will be doing t e walk t roug on t e model as seen in igure starting wit t e

creation of t e model in t e t roug to performing t e optimisation to estimate t eskeleton pose t as bones in total eac wit its own marker of t e bones areconnected b inges t at allow mo ement in onl a single a is and of t e bones areconnected using ball oints allowing complete freedom for rotations

ig : real world model

reating t e model

e a e pre measured t e offsets of t e bones and markers so all we need to do isinput t em into t e program e start b selecting reate a new model in t eo add a new bone click on t e parent bone to select it click on Add new bone

and t pe in t e name of t e new bone and t e offset from parent inall clickcreate to finis adding t e bone e model iewer will update to reflect t e

new bone added to t e model and t e new bone will automaticall become t e selected


bone Also note ou can c ange t e rotation orientation of an bone of t emodel b clicking on t e bone in t e iewer and ad usting t e sliders t at pop upon t e left of t e t is also possible to set t e constraints on rotation on t e boneb mo ing t e constraints marker on eac slider to t e appropriate alue Anot eruseful feature is t e bone ierarc iewer on t e rig t of t e it s ows t e parentof t e current bone and lists t e c ildren underneat as well licking on t e name ofan bone in t e bone ierarc will make t at bone t e selected bone

ig : reating a new model

op left tarting a new model op rig t selecting to add a new bone ottom left nputting t e

name and offset ottom rig t after adding t e tail bone

After creating t e complete skeleton model we will need to add markers in order fort e optimisation to recognise t e bone positions irst we will click on t e markermode button on t e top left of t e t is mode displa s as t e markers on t e models owing t e corner orientations and positions of eac marker o add a new marker toa bone select t e bone in t e iewer and t pe in t e offset of t e markerin t e input bo es on t e left of t e licking on Add new marker will add t emarker onto t e selected bone o modif t e rotation of a marker on a bone select


t e bone in t e iewer A list of markers attac ed to t at bone will appear on t e leftof t e licking on t e marker in t at list will select t e marker t is is reflectedb t e orange ig lig ting of t e selected marker in t e model iewer erotation sliders will appear on t e left e application will automaticall assign eacmarker an based on its internall created bone model tree t is possible to o erridet is be a ior and assign our own s to a marker b using t e te t input bo on t ebottom left w ilst a ing a marker selected en mo ing back to bone editor modet e corners of t e markers will still be isible on t e model

ig : Adding t e markers

op left Adding a marker op rig t elected marker and c anging orientation ottom arker

appearing bone mode

After we a e added all t e bones and markers we s ould sa e t e model o do t isclick on a e urrent odel in t e top rig t of t e A te t input bo will appeart pe in t e name ou wis to sa e t e model under and click sa e to confirm f we areo erwriting an e isting model a dialog bo will appear asking ou to confirm ou reallwant to o erwrite an e isting model ot erwise t e application will take a screens otof our model and sa e it into t e ocuments figurate models odel amefolder o load an e isting model click on t e oad e isting odel button on t e


top rig t of t e e will s ow displa pictures of all sa ed models simpl clickon t e model ou wis to load ou can also t pe t e name of t e model ou wantto load in t e te t input bo t e sa ed models t at a e a prefi matc ing t e t pedname will be ig lig ted in t e displa e sa ed model will be loaded wit all t erotations and offsets specified as w en ou sa ed it

ig : a ing and loading

op rig t a ing t e bug model op left oading t e same bug model

stimation of model pose

e a e completed creating t e model and placing t e markers in t e pre ious sectionwe will not go on to e plain ow to estimate t e model pose from a real world model

e rotated model we will be using for t is e ample is s ow in igure


ig : otated real world model

o start t e optimisation press e will c ange to s ow t e camera feed ont e rig t side and t e model iewer on t e left side ou can still use t e mouseto na igate and rotate around t e model during t e optimisation usable iewsa e to be collected before t e le enberg mar uardt kicks in As ou na igate t e

real world model wit t e camera t e camera feed will be o erlaid wit t e differencebetween e pected marker corners and seen seen marker corners based on t e dominantmarker in t e current iew is is to aid t e user in na igating t e model markerse ibiting a large difference s ould be prioritised for iewing wit t e camera ress

again to stop t e optimisation it is usuall clear w en t e estimated model matc est e real world model as t e e pected corners s ould matc up more or less wit t eactual corners in t e camera feed o erla


ig : ptimisation in action

igure s ows optimisation in action e top two images s ow t e optimisation int e beginning stages notice ow t e point difference is large on t e tail and wingsas t e optimisation is in t e process of estimating t ose parts of t e skeleton model

e bottom two images s ows t e optimisation towards t e later stages t e model int e model iewer is nearing t e rotations present in t e real world model ecorner seen and obser ed as almost completel matc ed up in t e o erla


ig : inal rotation model

igure s ows t e final estimated skeleton model pose after optimisation e resultsobtained are er accurate notice ow it is almost completel s mmetrical as it is int e real world model pon selecting t e bones and comparing it to t e real worldmodel rotations we see a similar sort of accurac to t e accurac obtained during t euantitati e anal sis testing e user can now sa e t is rotated model

inall t ere is a tool included w ic allows t e user to e port t is pose as a filewit a single frame of animation is allows t e user to import t e result into amodeling application suc as ma and attac a real model to t e estimatedpose for furt er use e user simpl as to click t e e port to button in t ebottom rig t of t e after optimisation as finis ed e file will be writtenout to t e same folder t at t e associated model is sa ed to


onclusion

roug out t is report we a e demonstrated an eas to use s stem t at allows a user togenerate a er accurate pose of a real world ob ect using ust a web cam in real time

is as been ac ie ed using computer ision tec ni ues in particular reconstructionfrom man iews using a e enberg ar uardt optimisation sc eme n t is sectionwill ig lig t a few important t ings to be taken from t is pro ect detail some of t elimitations of t e s stem and finall talk about some future work t at can be done inorder to impro e and add functionalit to t e s stem

ummar

sing bundle ad ustment tec ni ues can ield a er ig degree of accuract picall bone positions were less t an a few mm off and in t is case can alsoelp reco er from situations w ere not all t e feature points ma be isible in

t e scene or instance in t e situation of a ing a missing marker on a bonec ain between two seen markers we can t picall reco er t e rotation of t ebone wit t e missing marker based on t e information about t e markers eit erside of it An e ample of t is is s own in igurenitiall we were a ing problems wit some markers being mistaken for ot ersw en t e camera was not in focus or t e lig ting conditions were not ideal iscan cause problems as t e iew collected would be bad and ad ersel affect t eoptimisation e got around t is b sa ing t at a iew is not alid if t e markeris detected furt er awa t an t e skeleton model would allow from an ot ermarker is c eck is an n2) operation for markers in t e scene in practicet is is acceptable as e en if we ad t e full markers in t e scene t e timeto perform t is c eck would be minimalundle ad ustment is an n3) problem b nature using all t e information from

all t e iews simultaneousl would mean it would take far to long for real timeapplication e sol ed t is problem b splitting up t e tracking t e markersfrom t e optimisation of t e estimated rotations is is similar to t e approactaken b A urt ermore we intelligentl selected a limited amount of iewswe were using in order to make eac iteration of t e optimisation complete ina far faster time is reduced t e problem down to a nm problem witincreasing iew count


e also found t at doing a localised optimisation on certain parts of t e modelcan elps in speeding up t e o erall optimisation educing t e number of bonesin an optimisation pass reduces t e computation time e ponentiallnitiall we were running into t e problem of local minimums causing t e optimisation to terminate on an incorrect result e use of t e rotation estimateeuristic allows us to get around t is problem as well as speeding up t e process

of optimisation as we were starting closer to t e final solutiona ing t e o erla on t e web cam feed meant t at t e user can a e an eas

wa to see ow well t e optimisation is going t pro ed to be er useful inuickl determining if t e optimisation was working or not

ig : issing marker on wing notice ow t e final estimate reco ers t e pose of t e complete wing

imitations and future work

ne of t e ke limitations of t e s stem is t e need to accuratel measure t e modelto input into t e application a ing a badl measured model can ield bad resultsas t e optimisation would t ink t e rotations are off if it e pects t e markers to besomew ere else A furt er impro ement to get around t is would be to a e a selfcalibrating s stem w ic can determine t e lengt s of t e bones and possible rotationsust be looking at t e model wit t e web cam A possible wa to do t is would beto manuall take a number of iews wit t e bones rotated in a certain wa aroundt e real world model in order to find out w ere t e oints are on t e model

e application does not get around t e problem t at t e optimisation would still increasee ponentiall wit t e number of bones in t e model Alt oug it would probablbe unwise to use a different sc eme ot er t an e enberg ar uardt optimisation wecould c ange t e strateg related to ow man iews we use per run and t e c ances


of doing a local optimisation as opposed to a global one is strateg could be basedon an anal sis performed on t e model done before t e optimisation begins urt erwork is also possible on ow we rank t e iews t at we obtain for instance we canrank iews e ibiting large amounts of motion blur to be of worse ualit and baseour selection on more factors t an ust t e number of markers in a scene and t e mostdominant marker seen

a ing markers on t e real world model is still not t e most ideal solution A bigimpro ement could be made if we could make t e s stem marker less as well it wouldrel on real world features on t e model instead of t e fiduciar markers e wouldneed a good wa to be able to tell t e s stem w at to look for in t e scene and ow tocompre end t e features it finds in relation to t e skeleton model is would likelbe t e ne t step forward for t e application

Anot er are of future work would be to make t e application be able to detect w ent e real life model as c anged rotation in t e middle of an optimisation pass e couldt en be able to perform real time capture of t e c ange of rotation is can be furt erworked upon to make it retain information about t e bones t at a e not c angedrotation in order to make it determine t e new pose wit out a ing to recaptureinformation about t e w ole skeleton model

t would be useful if we could isualise t e full model t at will be used in programssuc as a in con unction wit t e estimated pose e could include t is in t e

iewer b allowing t e user to import a alid rigged model

inall it would be an interesting area to e plore if we could model bones t at a ec anging lengt s suc as springs or fabric ur optimisation would need to take t ec ange in si e of a bone into account as well as t e rotation is would allow us tomo e onto modeling not onl rigid skeleton ob ects but ob ects wit elastic propertiessuc as clot ing and ropes


Appendi AA ardware

e pro ect was de eloped and run on indows running on a ual core i gand gb of A e camera used t roug out t e pro ect was a non branded

web cam e camera intrinsic and e trinsic parameters are listed below:

o c a l l e n g t :r i n c i p a l p o i n t :i s t o r t i o n : − −

A oftware

e entire application was de eloped using t e programming language in isualtudio press edition e used icrosoft A for t e grap ics A Alglib for

t e e enberg ar uardt implementation and A A for t e marker detection eare using t e A A wrapper built from oblin A t is must be included as ain t e pro ect


ibliograp

oc kano erge Anatol e ic Alglib ttp: wwwalglib net

artle riggs c auc lan undle ad ustment a modern s nt esis page

rown ecentering distortion of lenses otometric ngineering:

isual target tracking

iala Artag a fiducial marker s stem using digital tec ni ues n rocomputer ociet onf omputer ision and attern ecognitionolume pages

inland Al ar ttp: irtual tt fi irtual pro multimedia al ar tml

aureen urniss otion capture

A Artoolkit ttp: www itl was ington edu artoolkit

artle and A isserman ultiple iew eometr in omputer isionambridge ni ersit ress : second edition

at ook ob arp ark inocc io ic ard oore Ale ipman Andrew lakeamie otton Andrew it gibbon eal time uman pose recognition in partsfrom single dept images

arlsson larke and A isserman etecting and tracking linear featuresefficientl

lein and urra arallel tracking and mapping for small ar workspacesn roc t and A nt mp i ed and Augmented ealit A

pages


adison io ision b ttp: researc cs wisc edu grap ics courses cseff b tml

onald ar uardt An algorit m for least s uares estimation of nonlinearparameters ournal of t e ociet for ndustrial and Applied at ematics :pp

ikon nderstanding focal lengt ttp: wwwnikonusa com learn ande plore p otograp tec ni ues g cu o o understanding focal lengt tml

berkampf e ent on and a is terati e pose estimation usingcoplanar points n roc omputer ociet onf omputerision and attern ecognition pages

ill riggs ilip c auc lan ic ard artle and Andrew it gibbonundle ad ustment a modern s nt esis n roceedings of t e nternationalorks op on ision Algorit ms: eor and ractice pagesondon pringer erlag

ang A fle ible new tec ni ue for camera calibration :

abstract - faculty of engineering | imperial college · pdf filef igurate: an application for...

Documents