Post on 10-May-2015
Embed Size (px)
DESCRIPTIONPipelines. Tpl dataflow. Use cases
- 1.Pipeline. TPL Dataflow. Usage.by Alexey Kursov http://www.linkedin.com/in/kursov
2. TPL Dataflow The Task Parallel Library (TPL) provides dataflow components to help increase the robustness of concurrency-enabled applications. These dataflow components are collectively referred to as the TPL Dataflow Library. Dataflow model providing inprocess message passing for coarse-grained dataflow and pipelining tasks... 3. WTF?Pipeline? Dataflow? 4. Pipeline basics In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, etc.), arranged so that the output of each element is the input of the next. Usually some amount of buffering is provided between consecutive elements. The information that flows in these pipelines is often a stream of records, bytes or bits.The concept is also called the pipes and filters design pattern. It was named by analogy to a physical pipeline. Simple example: 5. Pipeline basics A linear pipeline is a series of processing stages which are arranged linearly to perform a specific function over a data stream. The basic usages of linear pipeline is instruction execution, arithmetic computation and memory access.A non-linear pipeline (also called dynamic pipeline) can be configured to perform various functions at different times. In a dynamic pipeline there is also feed forward or feedback connection. Non-linear pipeline also allows very long instruction word. 6. Pipelines in real life 7. Pipelines in real life 8. Dataflow programming Dataflow programming is a programming paradigm that models a program as a directed graph of the data flowing between operations, thus implementing dataflow principles and architecture. emphasizes the movement of data program is series of connections explicitly defined inputs and outputs connect operations 9. Popular in parallel computing frameworks database engine designs digital signal processing network routing graphics processing 10. Usage In Unix-like computer operating systems, a pipeline is the original software pipeline: a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one. Each connection is implemented by an anonymous pipe. Filter programs are often used in this configuration. The concept was invented by Douglas McIlroy for Unix shells and it was named by analogy to a physical pipeline. Abstract and concrete examples: % program1 | program2 | program3 % ls | grep xxx 11. Usage Cascading is a Java application framework that enables typical developers to quickly and easily develop rich Data Analytics and Data Management applications that can be deployed and managed across a variety of computing environments. Cascading works seamlessly with Apache Hadoop and API compatible distributions. It follows a source-pipe-sink paradigm, where data is captured from sources, follows reusable pipes that perform data analysis processes, where the results are stored in output files or sinks 12. Usage Cascading pipeline example: 13. Usage Apache Crunch (Simple and Efficient MapReduce Pipelines by Cloudera) The Apache Crunch Java library provides a framework for writing, testing, and running MapReduce pipelines. Its goal is to make pipelines that are composed of many user-defined functions simple to write, easy to test, and efficient to run.Storm Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language 14. TPL Dataflow The Task Parallel Library (TPL) provides dataflow components to help increase the robustness of concurrency-enabled applications. These dataflow components are collectively referred to as the TPL Dataflow Library.Data Flow Tasks Coordination data structureTask parallel libraryThreads 15. What it provides for me? provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applicationsgives you explicit control over how data is buffered and moves around the systemimprove responsiveness and throughput by efficiently managing the underlying threadsallows you to easily create a mesh through which your data flowsmeshes can split and join the data flows, and even contain data flow loopsallows to create custom blocks and extend functionality 16. Type of blocks Dataflow blocks - are data structures that buffer and process data.1. source blocks (acts as a source of data ) ISourceBlock 2. target blocks (acts as a receiver of data) ITargetBlock 3. propagator blocks (acts as both a source block and a target block) IPropagatorBlock 17. Buffering blocks BufferBlock - stores a first in, first out (FIFO) queue of messages that can be written to by multiple sources or read from by multiple targets. If some target receives message from bufferblock, that message will be removed inputoutput (original)BroadcastBlock - broadcast a message to multiple components Currentinputoutput (originals or copies)TaskWriteOnceBlock - class resembles the BroadcastBlock class, except that a WriteOnceBlock object can be written to one time only inputFirst writed value (readonly)Taskoutput (originals or copies) 18. Execution blocks ActionBlock - is a target block that calls a delegate when it receives data input TaskTransformBlock - it acts as both a source and as a target and delegate that you pass should return a value of TOutput type inputoutput TaskTransformManyBlock - resembles the TransformBlock except that TransformManyBlock produces zero or more output values for each input value, instead of only one output value for each input value. inputoutput Task 19. Grouping blocks BatchBlock - combines sets of input data, which are known as batches, into arrays of output data. inputoutput TaskThe JoinBlock and JoinBlock - collect input elements and propagate out System.Tuple or System.Tuple objects that contain those elements input (T1) output input (T2)TaskThe BatchedJoinBlock and BatchedJoinBlock - collect batches of input elements and propagate out System.Tuple(IList(T1), IList(T2)) or System.Tuple(IList(T1), IList(T2), IList (T3)) objects that contain those elements input (T1) output input (T2)Task 20. LinkTo and Predicate Link/UnLink The ISourceBlock.LinkTo (returns IDisposable) method links a source dataflow block to a target block. If you want to unlink block you should call Dispose method on result of LinkTo call. The predefined dataflow block types handle all thread-safety aspects of linking and unlinking. Also the source will be unlinked automatically if you set MaxMessages larger than -1 on LinkTo call in DataflowLinkOptions after the declared number of messages is receivedPredicate When you link target block you can set predicate that will check message before adding it to input buffer. You should specify delegate in DataflowLinkOptions that recives message of TInput type of target block and returns bool value. 21. Another options You can specify:degree of parallelism for blockmaximum number of messages that may be buffered by the blocktask schedulernumber of message per taskcancellationgreedy behaviorcompletion 22. Recommendations Recommendations for building TPL Dataflow pipelines:make each block do one thing welldesign for compositionbe stateless where you can 23. Use cases 1.Prototyping pipelines for use in more complex systems2.Development of flexible asynchronous applications that process some data, like: Image processorsSound processorsPipelines in mobile phone appsData analysis/mining services 3.Web-crawlersetc.Study pipeline based development 24. Practice 25. Useful linkshttp://www.nuget.org/packages/Microsoft.Tpl.Dataflow/http://msdn.microsoft.com/en-us/library/hh228603.aspxhttp://blogs.microsoft.co.il/blogs/bnaya/archive/2012/01/28/tpl-dataflow-walkthrough-part-5.aspxhttp://www.cascading.org/http://crunch.apache.org/http://storm-project.net/ 26. Thanks for your attention!