a distributed framework for relaying stereo vision for telerobotics m. al-mouhamed, o. toker, a....

A DISTRIBUTED FRAMEWORK FOR RELAYING STEREO VISION FOR TELEROBOTICS

M. Al-Mouhamed,O. Toker, A. Iqbal, and M. Nazeeruddin

2

Contents

Introduction Distributed framework Video server Video client Server optimization Augmented reality (AR) AR support Telerobotic client framework Client-server framework performance evaluation Conclusion

3

Introduction

Telerobotics: humans to extend their manipulative skills over a distance, remote manual work, ..

Telerobotic applications Scaled-down: nano-scale, micro-surgery, surgery,

clean-room Hazardous: nuclear decommissioning & inspection,

disposal of dangerous objects, minefield clearance, operation in harsh environments like in space, underwater, ice, desert, ..

Safety: rescue, fire fighting,.. Security: surveillance, reconnaissance, .. Unmanned: oil platform inspection, repair, .. Teaching, training, entertainment, ..

4

Introduction … (cont.) Challenge: extending eye-hand motion coordination

through a network with high-quality perception, dexterity, and intelligent computer aided teleoperation. In natural eye-hand motion coordination, operator sees his hand

and reacts accordingly. Telerobotics:

Operator holds a master arm to dictate his hand motion, Motion is transmitted to a remote slave arm and reproduced (replica), Operator wears a head-mounted display (HMD) to see in 3D the effects

of his motion on the remote tool, Operator does not see his hand (HMD) nor the master arm, his hand is

logically mapped to the remote tool, Operator logically acts on the remote tool seen through the HMD.

Stereo vision: 3D perception of remote scene, a metric to calculate 3D position and orientation of objects, a tool to augment the real space (augmented reality), ..

5

A reliable and portable client-server frame work for Computer Aided Telerobotics (CAT)

Real-time streaming of video data from server to client Multithreaded system using Visual C#, DirectX, and Windows sockets

for efficient capturing and relaying of video DirectX, COM interfaces for graphic functionalities like DirectShow Client support to AR, e.g. graphic modeling, superimposing animated

graphics, etc. Support CAT operability, e.g. tool definition, activation of server

automation tasks, indexing function, space scalability, shared control, impedance control, etc.

HumanOperator

Internet

Master Arm, HMD,Glove,

Force feedback

Sensors, Video,Sound, Force

FeedbackActuators,

Hands, Arms,Feet

Distributed framework

6


Development of a client-server framework for grabbing and relaying of stereo video

Video server: DirectShow for effective interfacing, capturing, and

playback of video data Capture or grab stereo images from two digital

cameras (FireWire interface) Establish a reliable client-server connection Upon requests from the client send the stereo frames

(two pictures) to the client through windows sockets Camera control, disparity, focus, orientation, etc

7


Robot

VisionServer

Internet

DecisionServer

Server UI

PUMAComponent

UnimationController

Force Component

DigitalCameras Horizontal disparity = 6 cm

N

8


Video client: Detect and establish the connection with server Establish a fast graphic display system to show the

pictures received from the server Display the pictures arrived from the server and

continue in a loop each time asking a new stereo frame from the server

Allow the viewer to adjust the alignment of pictures on the HMD to compensate for the misalignment and non-linearity present in the camera at server

Control server camera setup

9


(Video client) Augmented reality: superimpose graphics (virtual)

over the stereo views of real scene to augment the stereo space

Avoid stop-and-wait in delayed teleoperation At client, emulate a sequence of simple actions over current

state of server robot using a model of environment with interactions

Send a converging solution for execution at remote robot Supervise real execution while receiving feedback

Seamlessly adds virtual (registered) objects into the real scene to avoid any perplexities for the teleoperator

Carry out task planning (non-invasive) Pre-run automatic tasks (tool switching, imp. control,..)

10


Internet

VisionClient

Decision ServerInterface

Client UserInterface

MasterArmComponent

Master

Arm

StereoDisplay

Head-Mounted Display,Eye-Shuttering Glasses

11

Video server

Network services like send/receive for video/data using windows sockets

Stereo video setup uses synchronous windows sockets SampleGrabber (DirectShow) to capture video frames

coming through a stream from a stereo camera setup Two schemes to transfer the video data, usage of

multiple threads, optimization to reduce network traffic General overview of the image grabbing and displaying

system

12

Video server

13

Video client GDI (Graphics Device Interface) is used to display the

received pictures from the server

14

Client framework

WindowsSocketsStereoSocketClient

Public Methods

Connect() StartReceivingStereo() StopReceivingStereo() Disconnect()

PublicProperties

LeftImage RightImage StereoImage RemoteHost RemotePort

Public Events

StereoFrameReady

Internet

15

……………..

Server optimization(single buffer, serialized transfer)

Internet

internet

…………..

16

Server optimization(double buffer, concurrent transfer)

Optimize the transfer of video data using thread engineering on the server

Concurrent thread execution among capture and transfer threads is achieved using double buffers on the server side

The thread responsible for sending the video data over the network will not wait after receiving a request for a picture from the client

Limited overhead in the allocation of extra buffer in the server DRAM and management

Pipelined execution: higher stereo frame rate due to time overlap between capture thread and network transfer thread

Need independent hardware for capture and transfer paths

17

Server optimization(double buffer, concurrent transfer)

………..

Internet

18

Augmented reality (AR)

Camera model A one-to-one mapping of scene and graphics Weak-perspective camera model Affine invariant projection for 2D-3D mapping Identify two projection matrices (left and right)

Microsoft DirectX APIs for developing 2D and 3D graphics Graphics drawn on a DirectX surface (access to pixels) HAL provides a set of graphics functions on hardware devices Primary surface is the current buffer, next data frame to off-screen

secondary surface, and use of page flipping Server:

Setting up cameras Placement and removal of reference frame

Client: StereoSocketClient component IdentifyCamera component RobotModel component DXInterface component

19

AR support

20

Telerobotic client framework

DecisionServerComponent

INTERNET

StereoSocketClientComponent

DXInterfaceComponent

HMDController

StereoVideo

rl MM ,

IdentifyCameraComponent

Left, RightImages

DecisionServerInterface

MasterArmComponent

),( MX Master

Arm

RobotModelComponent

AugmentedVideo

Vision Server

),( newnew MX

),( MX

21

Video Client-Server FrameworkPerformance Evaluation

Testing on University campus network Client-server P-4, 2 GHz, 100 Mbps NIC A video picture is 288x360 pixels, a pixel is 3 bytes, a picture

is 304 KB, a stereo is 0.6 MB, a stereo frame needs 5 Mbps bandwidth

Metrics: quality of depth perception, time delays, etc. Video clips on teleoperation tasks like peg-in-hole insertion,

assembly of a small water pump, wire-wrapping, pouring water, ..

22


Copying from SampleGrabber to DRAM

Case 1.1: A single thread to copy

stereo frame from SampleGrabber to DRAM on server

300 stereo frames Mean value = 24.025 ms 95% CI between 23.29 ms

and 24.75 ms.

23



Case 1.2: First thread to copy stereo

frame from SampleGrabber to DRAM on server

Second thread to read a force data without transfer

300 stereo frames Mean value = 60.48 ms 95% between 8 ms and

150 ms.

24



Case 1.3: First thread to copy stereo

frame from SampleGrabber to DRAM on server

Second thread to read a force data with transfer

300 stereo frames Mean value is 33.46 ms 9.43 ms additional for

adding network transport thread.

Need thread management

25


26


A magnified plot of inter-arrival times of force in the presence of stereo video and command streams.

27


Transferring over the campus network

Case 2.1: Single video thread Single buffer Serialized transfer

300 stereo frames Mean value = 86.1 ms 11.61 stereo

frames/second.

28


Transferring over the campus network

Case 2.2: Single video thread Double buffer Concurrent transfer

60,000 stereo frames Mean value = 58.94 ms 17 stereo frames/second. 90% CI between 56.0 and

64.8 ms.

29

Video Client-Server FrameworkSummary

Scheme Cameras to server DRAM (ms)

Server to client (ms)

Frames per second

Single buffer, serialized

24.025 86.1 11.61

Double buffer, concurrent

24.025 58.94 17

17 uncompressed stereo fps of size 360X288 pixels

30

Conclusion

A stereo video client-server framework using Multi-threaded programming, Visual C++ and Visual C#.NET programming languages

Reliable connection between client and server Visualization support to eye-shuttering and HMD page flipping A mathematical framework to support generation of complex

geometrical shapes and interactive AR AR teleoperation tools to overcome network delays and reduce

needed bandwidth A stereo video transfer rate of 18 fps over University campus

network Excellent depth perception using HMD techniques even for scaled-

down operations

a distributed framework for relaying stereo vision for telerobotics m. al-mouhamed, o. toker, a....

Documents

stereo video video server

distributed framework

remote tool

video directx

hand hmd

nazeeruddin slide

directshow client support

relaying stereo vision