1

33rd 33rd AnnualAnnual Simulation Simulation symposium symposium ANSS2000, April 16-20, 2000. Washington, D.C ANSS2000, April 16-20, 2000. Washington, D.C

Samir OtmaneSamir Otmane E-mail : otmane@cemif.univ-evry.fr E-mail : otmane@cemif.univ-evry.fr

CEMIF, Laboratoire Systèmes ComplexesCEMIF, Laboratoire Systèmes Complexes Http : http://lsc.cemif.univ-evry.fr:8080/~otmane Http : http://lsc.cemif.univ-evry.fr:8080/~otmane 40 Rue du Pelvoux 91020 Evry, France40 Rue du Pelvoux 91020 Evry, France Tél Tél : 01/69/47/75/04 Fax : 01/69/47/75/99 : 01/69/47/75/04 Fax : 01/69/47/75/99

Active Virtual Guides as an Active Virtual Guides as an Apparatus for AugmentedApparatus for Augmented

Reality Based Telemanipulation Reality Based Telemanipulation System on the InternetSystem on the Internet

Active Virtual Guides as an Active Virtual Guides as an Apparatus for AugmentedApparatus for Augmented

Reality Based Telemanipulation Reality Based Telemanipulation System on the InternetSystem on the Internet

2

ContentsContents

IntroductionIntroduction ARITI SystemARITI System Virtual guides/fixtures Virtual guides/fixtures

• Unified FormalismUnified Formalism

• Simple and Complex Virtual fixturesSimple and Complex Virtual fixtures

• Virtual guides representationVirtual guides representation

• Manipulation on the screenManipulation on the screen

• Deforming guides to generate an appropriate guidesDeforming guides to generate an appropriate guides

Experiments and ResultsExperiments and Results Conclusion and perspectivesConclusion and perspectives

3

IntroductionIntroduction- Tele-work -- Tele-work -

Master site Master site Communication supportCommunication support Slave site Slave site• The slave site is distant from the master site.The slave site is distant from the master site.• Information feedback is corrupted by a bandwidth Information feedback is corrupted by a bandwidth

limitation of communication support .limitation of communication support .• Time delay is not constant when using any Time delay is not constant when using any

communication network.communication network.• No portable and user-friendly Tele-work systems.No portable and user-friendly Tele-work systems.• Human performances are decreased during direct control Human performances are decreased during direct control

of remote Tele-manipulation task.of remote Tele-manipulation task.

Sending Sending ordersorders

Information Information feedbackfeedback

4

Virtual realityVirtual reality and and Augmented RealityAugmented Reality technologies are used technologies are used to :to : Overcome the instability of time delay,Overcome the instability of time delay, Complete or compensate the information feedback (video Complete or compensate the information feedback (video

feedback for instance) feedback for instance)

JAVA JAVA programming Language is used to implement the Man programming Language is used to implement the Man Machine Interface of Machine Interface of ARITIARITI system to : system to : Give a portable system and Give a portable system and User-friendly Tele-work systemUser-friendly Tele-work system

ARITIARITI system system

5

Interaction between Human Interaction between Human and and remoteremote Task Environment Task Environment

During interaction control of a remote robotic terminal tool, the user mustDuring interaction control of a remote robotic terminal tool, the user must

• Perform a physical action to initiate motion from the robot,Perform a physical action to initiate motion from the robot,

• Wait for the system to respond,Wait for the system to respond,

• Perceive the physical effect onto the robot and task environment,Perceive the physical effect onto the robot and task environment,

• Decide what to do next,Decide what to do next,

• Repeat the cycle until the task is completed.Repeat the cycle until the task is completed. Motor activity is initiated through Motor activity is initiated through interactioninteraction with a software interface via with a software interface via

Keyboard and mouse, joystick, master arm, etc...Keyboard and mouse, joystick, master arm, etc...

HHAARRDD

SSOOFFTT

NN

EE

TT

WW

Human Human OperatorOperator

Robotic InterfaceRobotic Interface Remote Robotic EnvironmentRemote Robotic Environment

6

Interactions withInteractions withA R I T IA R I T I

Three kinds of visual assistance are given to human Three kinds of visual assistance are given to human operator for operator for friendly human computer interactionfriendly human computer interaction using the using the ARITIARITI interface. These visual helps are devoted to : interface. These visual helps are devoted to : Environment perceptionEnvironment perception Robot controlRobot control Robot supervisionRobot supervision

PerceptionPerception

ControlControl

SupervisionSupervision

Human Human OperatorOperator

HHAARRDD

NN

EE

TT

WW

Remote Remote EnvironmentEnvironment

Robotic InterfaceRobotic Interface

7

Assistance for Assistance for EnvironmentEnvironment PerceptionPerception

Several Virtual view pointsSeveral Virtual view points

++

Video image feedbackVideo image feedback

Perception modulePerception module

Human Human OperatorOperator

In Control moduleIn Control module

8

Assistance for Assistance for RobotRobot SupervisionSupervision

Textual information of the Textual information of the current task current task

++Overlaid Model / ImageOverlaid Model / Image

Supervision moduleSupervision module

Human Human OperatorOperator

9

Assistance for Assistance for Robot ControlRobot Control

Virtual robotVirtual robot

Control moduleControl module

Human Human OperatorOperator

(HO)(HO)

In Supervision moduleIn Supervision module

10

System descriptionSystem description Hardware Hardware

ARITI system is implemented on a PC Pentium 233 Mhz with a 128 ARITI system is implemented on a PC Pentium 233 Mhz with a 128 Mo RAM.Mo RAM.

The PC is equipped with a Matrox Meteor video acquisition card The PC is equipped with a Matrox Meteor video acquisition card connected to a black and white camera.connected to a black and white camera.

The orders are sent via the RS232 serial link.The orders are sent via the RS232 serial link.

RS232 serial linkRS232 serial link

OrdersOrders Video acquisitionVideo acquisition

11

System descriptionSystem description SoftwareSoftware

ARITI system is implemented under LINUX operating system.ARITI system is implemented under LINUX operating system. ARITI interface is written based on JAVA object programming ARITI interface is written based on JAVA object programming

language language Video server is written using the C standard language.Video server is written using the C standard language. Control server is written using the C and ASM (Microprocessor Control server is written using the C and ASM (Microprocessor

Assembly Language )Assembly Language )

Video ClientVideo Client

Control ClientControl Client

-ARITI- -ARITI- INTERFACEINTERFACE

Applet JAVAApplet JAVA

L I N U X - O S -L I N U X - O S -

Video serverVideo server

Control serverControl server

C and ASMC and ASM

SS

OO

KK

EE

TT

12

To use the To use the ARITI ARITI system system

A R I T I A R I T I SystemSystem

CameraCameraRobotRobot

WWW CLIENTS + Internet BrowserWWW CLIENTS + Internet Browser

13

The The ARITIARITI Display Display

http://lsc.cemif.univ-evry.fr:8080/Projets/ARITIhttp://lsc.cemif.univ-evry.fr:8080/Projets/ARITI

14

Question !!Question !!

How to increase How to increase Human Human Operator performancesOperator performances to to do Telemanipulation task do Telemanipulation task

very easier ?very easier ?

15

IncreaseIncrease Assistance for Assistance for Robot ControlRobot Control

Virtual robotVirtual robot

++

Virtual FixturesVirtual Fixtures

Control moduleControl module

Human Human OperatorOperator

In Supervision moduleIn Supervision module

16

Virtual Fixtures StructureVirtual Fixtures Structure

Name Identify the fixture

Type Simple or complex and active or passive. If complex, itcontains the different links to the combined fixtures

Referential Contains position and orientation of the fixture on X, Yand Z axis

Attachment Static or dynamic and contains the coordinates of thevirtual point or object where fixture will be attached

Effect zone Contains the equation of the volumetric form, surface orother any known geometric shape may be associated tothe VF

Pre-condition Contains the activation condition of fixture

Function Contains a set of actions to be performed inside thevirtual guide either by the robot, by the HO or by bothactors

Post-condition Contains the inactivation condition of fixture

17

Simple Virtual FixturesSimple Virtual Fixtures

Human operator can create and use virtual fixtures to Human operator can create and use virtual fixtures to control the robot very easier.control the robot very easier.

Some examples of simple Virtual Fixtures (VF) :Some examples of simple Virtual Fixtures (VF) :

DiscDisc SphereSphere PlanPlan

Super-ellipsoid Super-ellipsoid ConeCone

CylinderCylinder PipePipeCube or Cube or SquareSquare

18

Complex Virtual FixturesComplex Virtual Fixtures

Delimiting the workspace between two robots in cooperation

Following an arbitrary trajectory by the robot

Reaching a dangerous target with the robot end tool

19

Method of Method of construction construction

taking some significant points on the surface of the guidetaking some significant points on the surface of the guide joining these points in order to get a wire frame joining these points in order to get a wire frame

representationrepresentation

Example :Example : • If a parametric equation of the guide is :If a parametric equation of the guide is :

• withwith

• Then the vertexs are :Then the vertexs are :

• And the segments are :And the segments are :

20

Manipulation onManipulation onthe screenthe screen

Use of the graphic camera model Use of the graphic camera model • Is the matrix Is the matrix MM which transform which transform point coordinates (Xo, point coordinates (Xo,

Yo, Zo ) Yo, Zo ) In the referential In the referential Ro,Ro, onto the screen point onto the screen point coordinates coordinates (U, V)(U, V) . .

21

Selection of the Selection of the 3D Fixtures on 3D Fixtures on

the screenthe screen

The designation on the screen = 2D point (U, V)The designation on the screen = 2D point (U, V) Determining what object 3D wanted to designateDetermining what object 3D wanted to designate The designated point belongs to a The designated point belongs to a DD segment, segment,

witch equation is :witch equation is :

22

Selection of the Selection of the 3D Fixtures on 3D Fixtures on

the screenthe screen

Determining what vertex on the 3D objects having Determining what vertex on the 3D objects having the smallest distance from the line segment D. the smallest distance from the line segment D.

Selected pointSelected point ScreenScreen

Selected Selected fixturefixture

23

DeformingDeformingVirtual FixturesVirtual Fixtures

Each virtual fixtureEach virtual fixture is associated to a graph is associated to a graph

where where XX is a set of vertexs, and is a set of vertexs, and UU a set a set of lines .of lines .

We define an application We define an application VV which associates any which associates any vertex vertex xx in in X X a set of his neighbors :a set of his neighbors :

We call We call a distance a distance between the vertex between the vertex xx and and yy..

24

DeformingDeformingVirtual FixturesVirtual Fixtures

If If xx00 is a start point of deformation andis a start point of deformation and 0 0 the value of this deformation, the value of this deformation, Then the value of deformation of the fixture is Then the value of deformation of the fixture is

given by :given by :

Where Where pp is called is called initial propagation factorinitial propagation factor and and ff is called the is called the dissipation of propagation dissipation of propagation

factorfactor

25

DeformingDeformingVirtual FixturesVirtual Fixtures

P = 0.99, f = 0.9P = 0.99, f = 0.9P = 0.99, f = 0.99P = 0.99, f = 0.99

P = 0.99, f = 0.9P = 0.99, f = 0.9

26

Experiments Experiments

Pick and place taskPick and place taskTele-operation modeTele-operation mode Control the real robot via the virtual robotControl the real robot via the virtual robot10 human operators (HO)10 human operators (HO)3 kinds of test3 kinds of test

Without Virtual FixturesWithout Virtual Fixtures With With passivepassive Virtual FixturesVirtual Fixtures With With active (attractive)active (attractive) Virtual Fixtures Virtual Fixtures

Each HO makes 10 tests for each kindEach HO makes 10 tests for each kind

27

Experiments Experiments Task BoardTask Board

The robot is assumed to assemble (place) and disassemble The robot is assumed to assemble (place) and disassemble (pick) objects hanging on a metal stand(pick) objects hanging on a metal stand

ObjectsObjects

Metal Metal standstand

Head of the Head of the robot pegrobot peg

TargetTarget

28

Fixture to reachFixture to reacha targeta target

A simple geometric primitive :A simple geometric primitive :• Cone .Cone .

29

Fixture to pick theFixture to pick theobject and unhook itobject and unhook it

A complex Virtual guideA complex Virtual guide• Combining a 3 simple guides (cylinders)Combining a 3 simple guides (cylinders)

30

Fixture to place theFixture to place the object onto the stand object onto the stand

A complex Virtual guideA complex Virtual guide• Combining 4 simple guides ( 1 cone + 3 cylinders)Combining 4 simple guides ( 1 cone + 3 cylinders)

31

ResultsResultsReach a cylinder N° 1Reach a cylinder N° 1

Reach a 3D target point on the peripheral of the cylinder N°1Reach a 3D target point on the peripheral of the cylinder N°1 Without virtual fixtures there is Without virtual fixtures there is 1.491.49 collision for each test collision for each test

32

ResultsResultsReach a cylinder N° 1Reach a cylinder N° 1

Imprecision errors on X, Y, Z axisImprecision errors on X, Y, Z axis

Err < 0,25 mm Err < 0,25 mm with active VFswith active VFs

Average timeAverage time

7,7 sec7,7 sec with active VFs with active VFs

33

ResultsResults Pick and place a cylinder N° 1Pick and place a cylinder N° 1

- Blue - with passive VFs.- Blue - with passive VFs. - Red - with active (attractive potential fields) VFs.- Red - with active (attractive potential fields) VFs.

Unhook a cylinder N° 1 Unhook a cylinder N° 1 Passive VFs - Passive VFs - 12,78 sec12,78 sec Active VFs - Active VFs - 9,5 sec9,5 sec

Place a cylinder into the stand, Place a cylinder into the stand, Passive VFs - Passive VFs - 37,96 sec37,96 sec

Active VFs - Active VFs - 7,86 sec7,86 sec

34

35

36

37

ConclusionConclusion

Thanks to Virtual Fixtures the human operator Thanks to Virtual Fixtures the human operator performances are increased :performances are increased :

best accuracy < 0,25 mmbest accuracy < 0,25 mm best completion timebest completion time best safetybest safety

JAVA programming Language is used to JAVA programming Language is used to implement these Virtual Fixtures into the implement these Virtual Fixtures into the ARITIARITI system to :system to : Give a portable and interactive fixtures.Give a portable and interactive fixtures.

38

PerspectivesPerspectives

Use the Use the Virtual FixturesVirtual Fixtures for mobile robot application, such as for mobile robot application, such as

• Navigation, obstacles avoidance, to assist disable person.Navigation, obstacles avoidance, to assist disable person.

39

PerspectivesPerspectives

Add Tactile functions to activate fixtures Add Tactile functions to activate fixtures Extend the use of virtual fixtures to do a cooperative Tele-Extend the use of virtual fixtures to do a cooperative Tele-

WorkWork

Extended Extended A R I T I A R I T I SystemSystem

User 1User 1

User 2User 2

User nUser n

..

..

..

Robot 1Robot 1

Robot 2Robot 2

Robot mRobot m

..

..

..

N N E E TT