supporting a volume rendering application on a grid-middleware for streaming data
DESCRIPTION
Liang Chen Gagan Agrawal Computer Science & Engineering Ohio State University. Supporting a Volume Rendering Application on a Grid-Middleware For Streaming Data. Introduction- Motivation. What is data steam Data stream: data arrive continuously - PowerPoint PPT PresentationTRANSCRIPT
1
Supporting a Volume Rendering Application on a Grid-Middleware
For Streaming Data
Liang Chen Gagan Agrawal
Computer Science & Engineering Ohio State University
2
Introduction-Motivation• What is data steam
– Data stream: data arrive continuously – Enormous volume and must be processed
online– Need to be processed in real-time– Data sources could be distributed
• Data Stream Applications:– Online network intrusion detection– Sensor networks– Network Fault Management system for
telecommunication network elements
3
Introduction-MotivationNetwork Fault Management System (NFM)
analyzing distributed alarm streams
Switch Network
X
NFM (Network Fault Management) System
4
Introduction-Motivation
Switch Network
X
• Challenges– Data and/or computation intensive– System can be easily overloaded
5
Introduction-Motivation • Possible solutions
– Grid computing technologies– Automatically adjust processing rate
Switch Network
6
Introduction-Our Approach• We implemented a middleware to meet the nee
ds• Previous work:
1. Utilizing existing grid standards Liang Chen, K. Reddy and G. Agrawal “GATES: A Grid-Based Middleware for
Processing Distributed Data Streams”.HPDC, 2004. 2. Providing self-Adaptation functionality Liang Chen and G. Agrawal “Supporting Self-Adaptation in Streaming Data M
ining Applications”. IPDPS, 2006.
3. Supporting automatic resource allocation Liang Chen and G. Agrawal “A Static Resource Allocation Framework for Gri
d-Based Streaming Applications”. Concurrency Computation: Practice and Experience Journal, Volume 18, Issue 6 , Pages 653 - 666.
4. Supporting efficient dynamic migration Liang Chen, Q. Zhu and G. Agrawal “A Supporting Dynamic Migration in Tig
htly Coupled Grid Applications”. SC 2006.
7
Roadmap• Introduction• GATES Overview • Adaptive Volume Rendering• Conclusions
8
GATES Architecture and Design
• Use Globus Toolkit, built on OGSA
• Allows users to specify their algorithms implemented in Java
• Take care of plugging user-defined algorithms into the system and running them in Grid.
• Applications need be broken down into a number of pipelined stages
9
A B C
Stage A Stage B Stage C
:GATES services
:Stages of an application :Queues between Grid services
:Buffers for applications
System Architecture and Design(Architecture)
Application
Stage A
Stage B
Stage C
10
Public class Second-Stage implements StreamProcessing{ … void work(buffer in, buffer out) {
… while(true) { DATA = GATES.getFromInputBuffer(in); Inter-Results = Processing(Data); GATES.putToOutputBuffer (out, Inter-Results); }
}}
System Architecture and Design
(GATES API Functions)
11
Adaptation Parameter• Definition:
– A parameter in an application– Changing the parameter’s value can change
processing rate of the application, also impact accuracy of the processing
• Two kinds of adaptation parameters– Performance parameter– Accuracy parameter
– Example• Sampling rate is an accuracy parameter
AccuracyProcessing rateAccuracy Parameter
AccuracyProcessing ratePerformance Parameter
12
Pseudo Codes Again with Self-adaptation API Functions
Public class Second-Stage implements StreamProcessing{ … //Initialize sampling-rate Sampling-rate = (Max+ Min)/2; void work(buffer in, buffer out) {
GATES.specifyAccuracyPara(Sampling-rate, Max, Min);
while(true) { DATA = GATES.getFromInputBuffer(in); Inter-Results = Processing(Data, Sampling-rate); GATES.putToOutputBuffer (out, Inter-Results); Sampling-rate = GATES.getSuggestedValue(); }
}}
13
Adaptive Volume Rendering
• Motivation – Grid computing is needed• Visualization involves large volumes of dataset • We focus on streaming volume data• Interactively visualizing volume data in real-time is
needed– Computationally intensive– Resources consumed– Real-time processing can not be guaranteed
• The places where data are generated are distributed
• Typical client-server architecture is not scalable– Network bandwidths of wide-area networks are low– Computing capability of normal desktop is not enough
• Grid techniques would be a good solution– Divide the procedure into stages organized in a
pipeline – Allocate nodes close to data source to pre-process
volume data– The size of intermediate results is much smaller
14
Adaptive Volume Rendering
• Motivation – GATES is desirable– Automatic adaptation is desirable
• Volume rendering algorithms running on a grid need to be highly adaptive
• Adaptation usually achieved by manually adjusting adaptation parameters
• Such manual parameter adaptation is very challenging in a grid environment
– Automatic resource allocation is desirable• Grid environment is highly changeable
– The GATES middleware could fulfill the needs• Grid-based• Provide the self-adaptation function to applications• Automatically allocate Grid resources
15
• Overall design– Two pipelined steps – the first step:
• Build octrees from volume data– Octree is a tree data structure, in which each internal no
de has up to 8 children– Here, we use an octree to represent multiresolution info
rmation for a volume– Procedure to build an octree for a volume is as follows:
» Divide volume space into 8 subvolumes and create 8 children nodes
» For each subvolume, calculate standard deviation of all voxels in the subvolume, and store the deviation to the corresponding child node
» If the deviation is larger than a pre-defined value, divide the subvolume, repeat the above procedure. Otherwise, stop
Adaptive Volume Rendering
16
Adaptive Volume Rendering
• Overall design– Two pipelined steps – the second step:
• Use an octree and its corresponding volume to render images
• Provided an error tolerance (or user-defined resolution), use DFS to traverse the octree and stop at the nodes where the deviation is less than the resolution or error tolerance.
• Project the corresponding 3D-subvolumes to an image
17
Adaptive Volume Rendering
18
Adaptive Volume Rendering
• Make the rendering self-adaptive
– Two adaptation parameters used in the third stage• Error Tolerance – performance parameter• Image Size – accuracy parameter
– Only one adaptation parameter can be adjusted by GATES. So we fix one and adjust the other
19
20
Adaptive Volume Rendering
• Experiment 1
21
Adaptive Volume Rendering
• Experiment 2
22
Adaptive Volume Rendering
• Experiment 3: compare the performance of two implementations– Java-imple– C-imple
23
Conclusion• Grid computing could be an effective
solution for distributed data stream processing
• GATES – Distributed processing– Exploit grid web services– Self-adaptation to meet the real-time
constraints– Grid resource allocation schemes and
dynamic migration