cbware - distributed middleware for autonomous ground vehicles
DESCRIPTION
CBWare - Distributed Middleware for Autonomous Ground Vehicles. Master’s Thesis Defense By Vidhyalakshmi Venkitakrishnan Advisor: Dr. Arun Lakhotia Center for Advanced Computer Studies University of Louisiana at Lafayette October 18, 2006. Presentation Outline. - PowerPoint PPT PresentationTRANSCRIPT
CBWare - Distributed Middleware for Autonomous Ground
Vehicles
Master’s Thesis DefenseBy
Vidhyalakshmi VenkitakrishnanAdvisor: Dr. Arun Lakhotia
Center for Advanced Computer StudiesUniversity of Louisiana at Lafayette
October 18, 2006
Presentation Outline
Part I - Motivation and Research Contributions
Part II - Background and Related Work
Part III - CBWare and Evaluation Results
Part IV - Conclusions and Future Work
Middleware in AGV’s - Motivation
I. Real-time Information Exchange
Processes are computationally intensive and complex
Need for distributing components on multiple machines to achieve fairness and efficiency
Middleware – Key to information exchange among distributed components
CajunBot
II. Sensor Data Fusion
LIDAR
(Range, Theta)
(x, y, z)Global (x, y, z)
INS
Sensor Data Fusion – contd.
LIDAR Queue
INS Queue
Fusing most recent reading from queues – not consistent
Both sources generate data at different frequencies
CajunBot on rough terrainDARPA GC 2005
Sensor Data in Rough Terrain
t1
III. Log Server
Need data from processes for post-analysis
Collect data to tune parameters
CajunBot – NQE 2005 Z spike - wall of obstacles detected from log data
IV. Remote Real-time Monitoring
Why Real-time monitoring?
Visual/graphical views of the world seen by system
Internal states of Obstacle Detection
Visualize live data during field testing
Debug problems with processes in real-time
Research Contributions
Real-time Information Exchange in distributed system
Support for fusion of consistent data
Log server
Real-time monitoring capability
Middleware Background
What is Middleware?
Layer of software that connects different application components.
Supports complex, distributed applications
Hides heterogeneity of underlying platforms
Types of Middleware
Publish/Subscribe
Client/Server
Remote Procedure Calls
Other IPC Mechanisms: Shared Memory, Message Queues, UNIX IPC
Publish/Subscribe Model
Powerful abstraction for distributed systems
Message-based anonymous communication
Publishers and Subscribers are decoupled
Good solution for scalability
Examples: CMU-IPC, NDDS, IBM-Gryphon, Siena
Publisher SubscriberPublish/Subscribe
MiddlewarePublisher
publishes data
Subscriber subscribes to data
Middleware
delivers data
Client/Server Model
Clients send request to the server
Server processes requests and responds back
Clients blocks until response from server
Client 1
Client 2
Client 3Server
Request
Response
Remote Procedure Calls
Protocol to extend Local Procedure
Involves two independent processes, which may reside on different machines Caller (Process A on Host A) issues procedure call to Callee
(Process B on Host B) with list of argument values Caller is suspended Callee executes procedure, returns values to Caller Caller resumes execution
Examples: Java RMI, CORBA, Microsoft DCOM
Robotic Middlewares
IPT (1996) Object-oriented, message passing toolkit Unmanned Ground Vehicle Client/Server Model
MIRO (2002) Middleware for Mobile Robots Based on CORBA
Broker (2005) IPC Toolkit for Multi-Robot Systems Works on Publish/Subscribe Model
CBWare Architecture
Works on Publish/Subscribe Model
• Two types of Interfaces CBQueues CBPackets
• Log server
• Remote monitoring
Dedicated Logging Machine
Separate Machine – Why? Hard disk – Most sensitive part of a machine Bumps – Common reason for hard disk
crash/failure Disk crash – Affects autonomous operations
Provision to log only sample of data to disk
Sampling data for remote monitoring
Why sample data sent over wireless network? Wireless network cannot handle all the data produced
Onboard network – 1 GB LAN Ethernet switch Wireless network – 802.11g Wireless
Bandwidth Ethernet LAN: 1000 Mbps 802.11g wireless: 54 Mbps
CBQueues
Interprocess communication on a single machine
Interface to read/write messages
Built using POSIX Shared memory
Shared Memory Model
Area of memory shared by multiple processes.
Shared Memory area -indistinguishable to a process from unshared memory
Code Code
Private data Private data
Shared data Shared data
Why Shared Memory Model?
Fastest form of IPC available
Negligible communication overhead
Need to deliver high bandwidth sensor data in a timely manner
CajunBot’s Shared Memory Model
Queues for every message type in shared memory area
Messages in every queue temporally ordered Crucial for Interpolation support
Single Writer Restriction
Only one writer for each message queue, no limit on the number of readers
Enables temporal ordering of data in distributed queues
Multiple producers for same message type Separate queue maintained for each producer
Data Fusion and Interpolation
INS generates data at 100 Hz Produces data at 10 ms interval
LIDAR LIDAR generates data at 75 Hz Scans at 13 ms interval
Most recent INS data may be up to 10 ms old when LIDAR scan is read
Stabilization of sensors
In CajunBot, no stabilization of sensors Fusing most recent data can give erroneous results 0.5 degree inaccuracy of angular difference -> 2 feet
displacement of global point from correct location Leverage rough terrain to increase visibility
In vehicles with sensor stabilization, can fuse most recent data Stabilization dampens rotating movements Sensors wont experience significantly different
orientations in the 10 ms period
CBPackets
Distributed interprocess communication across machines
UDP Broadcast
Support for multiple readers and writers
Why UDP Protocol?
Real-time applications Deliver messages in time
Small Packet Header overhead TCP Header = 20 bytes, UDP Header = 8 bytes
For a 20 byte message, compare: 40 bytes Vs 28 bytes
Example:Case 1: Message Size = 12 bytesEfficiency using TCP = (12 / 12+20) = 0.37 = 37%Efficiency using UDP = (12/ 12+8) = 0.6 = 60%
Case 2: Message Size = 1000 bytesEfficiency using TCP = (1000 / 1000+20) = 0.98 = 98%Efficiency using UDP = (1000 / 1000+8) = 0.992 = 99%
Why UDP Protocol? – contd.
Broadcast/Multicast capabilities Send messages to all processes at once
Connectionless nature Useful for remote monitoring No problems if wireless network fails
Data Marshaling
Why data marshaling? Distributed System – Machines with heterogeneous
architectures/OS follow different byte-ordering, alignment strategies
Data converted to neutral format for transmission over the network
Examples of Data Encoding standards: XDR, NDR, CDR
XDR used by CBWare
XDR Translation
Encode: Convert data from native format to XDR format
Decode: Convert data from XDR format to native format
Every message marshaled by CBWare before sending over network
Sender ReceiverXDR FormatEncode data Decode data
Monitoring Process Status
Processes on multiple machines generate status/warning/error messages
Monitoring messages on each machine Tedious task when system scales to higher level
Cbmesg: Interface for logging and real-time monitoring of process status information
Replication of Shared Memory
Combination of CBQueues and CBPackets
Subscriber replicates local shared memory Distribute queues to multiple machines Easy transfer of programs Scalability of computational power
CBWare Network Evaluation
• CBWare evaluation metrics End-to-End Latency (msec) Packet Rate Bandwidth (bytes/second) Packet Order
• Experimental setup• Dell Poweredge servers• 1 GB Ethernet LAN switch
CBWare Network Evaluation Results
End-to-End Latency Delays ranging from 0.4 msec to 6 msec depending on packet
size Transmission delay = Packet reception time – Packet sent time
CBWare Network Evaluation Results – contd.
Bandwidth and Packet Rate Packet Rate = (Bandwidth/Message Size) Maximum rate ranging from 1200 packets/second to 90
packets/second
Packet order Measurement based on timestamp No out of order packets on CBWare network No network congestion
Network Monitoring Tools used: Ethereal, tcpdump
Conclusion
CBWare – Middleware for Autonomous Ground Vehicle
Publish/Subscribe Model
Distributed Information exchange - Shared Memory and UDP Broadcast
Support for fusion of sensor data
Log server
Sampling for remote real-time monitoring
CBWare’s comparison with other Middlewares
CBWare CMU-IPC NML NDDS Broker IPT
Data interpolation &
Fusion
Yes No No No No No
Sampling data for Remote Monitoring
Yes No No No No No
Protocol Shared Memory +
UDP
TCP
UDP
Shared Memory + UDP
UDP UDP TCP Sockets
Languages C++ C++
Java
LISP
C++
Java
C++
Java
C++
Java
Python Perl
C++
Pub/Sub Models
Yes Yes Yes Yes Yes No
Future Work
Fault tolerance
Compression techniques for larger messages
Port CBWare to other Operating Systems
Questions
Extra slides
Distributed Computing System of CajunBot
Onboard Computing system of CajunBot Four machines NTP Separate networks
Why XDR?
XDR format of data is same on all machines Any machine can decode data encoded by
any machine Easily add machines with different
architectures
Log Control
Remote control of central log server (cb_logd) TCP Connection Communication using predefined ASCII Protocol
Three types of control signals Enable logging data Disable logging data Change log directory on log server
CBQueues Utilities
Remove corrupted shared memory: cb_qclean Clean shared memory area
Read/Write to shared memory: cb_qtest Testing and debugging problems with shared memory
CBWare’s Publish/Subscribe Components
Publisher: cb_publisher Handles socket operations Constructs CBPackets Marshal data Broadcast on network
Central Log Server: cb_logd Logs data received from broadcast network to disk Runs on separate machine to prevent disk failure affecting
autonomous operations Provision to log only a sample of data to disk Samples data at prescribed interval and broadcasts data on
wireless network for remote monitoring
CBWare’s Publish/Subscribe Components – contd.
Subscriber: cb_subscriber Runs in two modes
Mode 1: Replicate “shared memory” queues across machines Mode 2: Real-time monitoring on remote laptop
Easy transfer of programs to multiple machines
Easy scalability of computational power
CBQueues Interface
Read interfaces Most recent message Next message Check for new message
Write interface
CBPacket format CBPacket = CBHeader + Data
CBHeader = <Channel, Timestamp, Size, Checksum, Encoding Format>
Maximum transmission size upto 65507 bytes
CajunBot on flat terrain DARPA GC 2005
Sensor data in Flat Terrain
1 2 3 4 5 6 7 8 9 10 11
LIDAR Data
INS Data
Time Instance (t)
0
10
0
10
Pitch of the vehicle
(degrees)
Pitch of the sensor
(degrees)
t1