Processing Frameworks and Software SystemsIntroduction of Theme 4
Wei Wang
FuDan University
1
Discussions in Theme 4 Load Balancing Optimization for Data Processing on High
Throughput Computing Platform by Haihang You
Resources scheduling with Mesos in an astronomical datacenter by Shoulin Wei
Usage of evolutionary algorithms for optimum task schedulingin the SKA scale by Foivos Diakogiannis
Evaluating SKA-SDP Candidate Execution Frameworks (COTS)
by Wei Wang
2
Background
• SKA will produce huge data beyond the processing ability (2TB/s, 50PFLOPS)
• The data scale of SKA challenges network, I/O and computing power for the processing frameworks.
• Requirements of SKA-SDP data flow pipelines can not be satisfied by any existing processing framework.
3
Image satisfies scientific needs?
Grid
Grid
FFT
FFT
FFT Re-proj
Re-proj
Re-proj
Nsnapshots
Yes
Calibration
Grid
Big cycle: Adjust parameters to improve image qualitySmall cycle: Modify model image
deconvolution
Visibility Data
Sky Model
Imaging Pipeline
Image
Split 1
Split 2
Split n
Input
Output
Input
Characters of SKA-SDP Data Flow Pipeline• Even the smallest split can not be processed in a
machine for SKA scale.• Massive SDP data causes huge I/O pressure for the
processing frameworks. （It is better to move computing, not the data）
• SDP data flow pipelines needs iterative computing. To reduce I/O, the processing frameworks need to support distributed in-memory computing
• The processing frameworks need to have high availability, good scalability and good fault tolerance when handling numerous parallel tasks
5
Challenges of SKA-SDP• Challenge of Computing、Storage、 Network communication (System view)
– Data arrival rate：1-10 TB/s– Computing requirement：100PetaFLOP/s– Storage requirement： 300PB first buffer storage
• Challenge of resource and task scheduling , load balance(Processing framework view)– Heterogeneous resources integration– Heterogeneous tasks (CPU tasks, GPU tasks, …)– Task-level resilience
• Challenge of Real-time Imaging (Application view)– The data for the least imaging unit can not fit into memory of a machine (The data of the least
imaging unit: about 41.7 T for ska scale )– Current 1024*1024 image（Raw data is MB-GB level)
• Gridding and dirty processing ：1.307s• One clean iteration 1.65s
– SKA Scale 12000*12000 （Raw data is GB-TB level) time ?
Difficulties for Processing Frameworks
• Huge I/O costs
– Support distributed in-memory processing
– Avoid data transfer as possible as we can
– Distributed in-memory processing should be supported
• Huge Computing costs
– Support heterogeneous computing
– Need task and resource scheduling for heterogeneous environments
7
• Task concerning– Task execution style
• Basic Task Execution
• Event Driven Task Execution
– Task schedule• Static dataflow task
scheduling
• Resource aware scheduling
• Dynamic load-balancing
• Data locality aware scheduling
8
• Task concerning– Task resilience
• Task-level resilience
• Application-level resilience
– Task definition• Explicit data dependencies
• Nested parallelism
– Task execution• Granularity analysis
• Asynchronous execution
• Sequential memory consistency
Features of Functionality
Features of Functionality• Data concerning
– Data visit and data movement• Access to Storage
• Data movement between tasks
• Data re-arrangement between tasks
• Distributed Data-structures
• Explicit data movement
• Implicit data movement
– Data replacement• Automated data placement
• Memory management
9
• Platform concerning
– scalability• Heterogeneous scheduling
• Foreign function Interface
• Interoperability
• Supported architectures
– Platform power• Automated Programmer
Assistance
• Streaming Inputs into the EF
• Automatic data-dependency analysis
Candidates of SDP Processing Frameworks• StarPU
– Heterogeneous computing framework– Scalability and availability will be evaluated further
• Daliuge– A prototyping processing framework designed for SKA-SDP– Performance and availability will be evaluated further
• Spark– Proven universal distributed in-memory computing framework– Evaluation work for astronomical pipelines is going on
• Alluxio– A distributed in-memory file system
• ………
10
• Execution Framework Evaluation
• Evaluation of the functionality and the performance of commercial candidate systems
• Requirement Evaluation
• Analysis of the EF features needed by astronomical pipelines
• Resource evaluation needed by astronomical pipelines
• Decision of the candidate lists
• Benchmark publication
• Publication of benchmarks for the evaluation
COTS Introduction
11
• COTS Task tsk-8– Cots is led by Paul Alexander– Mainly investigate current mature commercial
systems– We start COTS investigation with Spark
• COTS baseline program on Spark– Bottleneck analysis
• All stages the execution time exceeds a minute include cogroup or groupByKeyoperations which cause shuffle operations
• Too many unnecessary copies by using “flatMap” operations
– optimization• Avoid shuffle operations by replacing
cogroup (broadcast, third-party key-value store serving as distributed memory storage)
• Using Alluxio to replace broadcast to avoid redundant memory usage
• The execution time of stage5 can be improved significantly by replacing the cogroup with the usage of the broadcast (broadcast the small data set )
• Stage 5 (29 minutes -> 51 seconds)• 1~13 stages (34 minutes -> 3 minutes)
– Future work• The utilization of GPU in Spark
Stage ID function
0 local_sky_model
1 extract_lsm
2 telescope_management
3 telescope_data
4 reprojection_predict_ifft
5 degridding_kernel_update_degrid
6 extract_lsm
7 phase_rotation_predict_dft_sum_visi
bilities
8 visibility_buffer
9 timeslots
10 phase_rotation_predict_dft_sum_visi
bilities
11 visibility_buffer
12 Solve
13 correct_subtract_visibility_flag
14 telescope_data
15 gridding_kernel_update_phase_rotation_grid_fft_reprojection
16 sum_facets
17 identify_component
18 source_find
19 println
Stage time Stage time Stage time Stage time
0 2s 5 29m 10 36s 15 3.6m
1 2s 6 2s 11 29s 16 1.3m
2 2s 7 53s 12 15s 17 1.3m
3 0.2s 8 45s 13 2.7m 18 0.05s
4 5.1m 9 1.7m 14 0.2s 19 0.05s
12
Optimization of Spark for Sdp-reduced Program
Conclusion
• A promising SKA-SDP processing framework should have the following features:
– Support of distributed in-memory computing
– Support of heterogeneous tasks and resources
– Support of dynamic task scheduling
– Support of hierarchical I/O and computing levels
13
Thanks for your attention!
14
