functional groovy

Dr Paul King

@paulk_asert

http:/slideshare.net/paulk_asert/functional-groovy

https://github.com/paulk-asert/functional-groovy

Functional Groov

Topics

Intro to Functional Style

• Functional Basics

• Immutability & Persistent Data Structures

• Laziness & Strictness

• Gpars & Concurrency

• Type Safety

• Word Split (bonus material)

• More Info

2006-2

Introduction

• What is functional programming? – Favour evaluation of composable

expressions over execution of commands

– Encourage particular idioms such as side-

effect free functions & immutability

• And why should I care?

– Declarative understandable code

– Reduction of errors

– Better patterns and approaches to design

– Improved reusability

– Leverage concurrency

2006-2

What makes up functional style?

• Functions, Closures, Lambdas, Blocks as

first-class citizens

• Higher order functions

• Mutable vs Immutable data structures

• Recursion

• Lazy vs Eager evaluation

• Declarative vs Imperative style

• Advanced Techniques – Memoization, Trampolines, Composition and Curry

• Compile-time safety

• Concurrency

Closures...

2006-2

def twice = { int num -> num + num } assert twice(5) == 10 assert twice.call(6) == 12 def twice10 = { 2 * 10 } assert 20 == twice10() def triple = { arg -> arg * 3 } assert triple(5) == 15 def alsoTriple = { it * 3 } assert alsoTriple(6) == 18 def quadruple = { arg = 2 -> twice(arg) * 2 } assert quadruple(5) == 20 assert quadruple() == 8 // ...

...Closures...

2006-2

// ... def callWith5(Closure c) { c(5) } assert 15 == callWith5(triple) def twiceMethod(int num) { num * 2 } assert twiceMethod(2) == 4 def alsoTwice = this.&twiceMethod assert alsoTwice(5) == 10 def alsoQuadruple = twice >> twice assert alsoQuadruple(5) == 20 def forty = quadruple.curry(10) assert forty() == 40 assert [10, 15, 20] == [twice, triple, quadruple].collect{ it(5) } assert 45 == [alsoTwice, alsoTriple, alsoQuadruple].sum{ it(5) }

...Closures

• Used for many things in Groovy: • Iterators

• Callbacks

• Higher-order functions

• Specialized control structures

• Dynamic method definition

• Resource allocation

• Threads

• Continuation-like coding

2006-2

def houston(Closure doit) { (10..1).each { count -> doit(count) } } houston { println it }

new File('/x.txt').eachLine { println it }

3.times { println 'Hi' } [0, 1, 2].each { number -> println number } [0, 1, 2].each { println it} def printit = { println it } [0, 1, 2].each printit

2006-2

Better Design Patterns: Builder

<html> <head> <title>Hello</title> </head> <body> <ul> <li>world 1</li> <li>world 2</li> <li>world 3</li> <li>world 4</li> <li>world 5</li> </ul> </body> </html>

import groovy.xml.* def page = new MarkupBuilder() page.html { head { title 'Hello' } body { ul { for (count in 1..5) { li "world $count" } } } }

• Markup Builder

...Better File Manipulation

2006-2

def out = new File('result.txt') out.delete() new File('..').eachFileRecurse { file -> if (file.name.endsWith('.groovy')) { file.eachLine { line, num -> if (line.toLowerCase().contains('groovy')) out << "File '$file' on line $num\n$line\n" } } }

...DSL example...

2006-2

show = { println it } square_root = { Math.sqrt(it) } def please(action) { [the: { what -> [of: { n -> action(what(n)) }] }] } please show the square_root of 100 // ==> 10.0

Inspiration for this example came from …

...DSL example

2006-2

// Japanese DSL using GEP3 rules Object.metaClass.を = Object.metaClass.の = { clos -> clos(delegate) } まず = { it } 表示する = { println it } 平方根 = { Math.sqrt(it) } まず 100 の平方根を表示する // First, show the square root of 100 // => 10.0

// source: http://d.hatena.ne.jp/uehaj/20100919/1284906117

// http://groovyconsole.appspot.com/edit/241001

interface Calc { def execute(n, m) } class CalcByMult implements Calc { def execute(n, m) { n * m } } class CalcByManyAdds implements Calc { def execute(n, m) { def result = 0 n.times { result += m } return result } } def sampleData = [ [3, 4, 12], [5, -5, -25] ] Calc[] multiplicationStrategies = [ new CalcByMult(), new CalcByManyAdds() ] sampleData.each {data -> multiplicationStrategies.each {calc -> assert data[2] == calc.execute(data[0], data[1]) } }

def multiplicationStrategies = [ { n, m -> n * m }, { n, m -> def total = 0; n.times{ total += m }; total }, { n, m -> ([m] * n).sum() } ] def sampleData = [ [3, 4, 12], [5, -5, -25] ] sampleData.each{ data -> multiplicationStrategies.each{ calc -> assert data[2] == calc(data[0], data[1]) } }

Language features instead of Patterns

Strategy Pattern

with interfaces

with closures

Topics

• Intro to Functional Style

Functional Basics

• GPars & Concurrency

• Type Safety

• More Info

2006-2

Pure Functions, Closures, Side-effects

2006-2

def x = 4 def increment = { arg -> arg + 1 } assert 11 == increment (10) assert x == 4 def incrementWithSideEffect = { arg -> x++; arg + 1 } assert 11 == incrementWithSideEffect(10) assert 101 == incrementWithSideEffect(100) assert x == 6

Referential Transparency

2006-2

def x, y, z, arg def method = { // ... } y = 3 arg = y x = y + 1 method(arg) z = y + 1 // z = x assert x == z def pythagorian(x, y) { Math.sqrt(x * x + y * y) } final int A = 4 final int B = 3 def c = pythagorian(A, B) // c = 5 assert c == 5

Show me the code

PrimesPalindromes.groovy, Composition.groovy, Memoize.groovy, FactorialTrampoline.groovy

Tail Recursion

• https://github.com/jlink/tailrec

2006-2

@TailRecursive def factorial(n, acc = 1) { n <= 1 ? acc : factorial(n - 1, n * acc) } println factorial(1000G)

Topics

Immutability & Persistent Data Structures

• Type Safety

• More Info

2006-2

Immutability • An object’s value doesn’t change once created

• Examples – Groovy has primitives & their wrapper classes,

Strings, null

– “constants” (final reference fields)

• With some caveats about what they point to

– Basic enum values

• With some caveats on complex enums

– Numerous (effectively) immutable classes • java.awt.Color, java.net.URI, java.util.UUID, java.lang.Class,

java.util.Date, java.math.BigInteger, java.math.BigDecimal – Your own carefully written classes

– Very careful use of aggregation/collection classes

– Special immutable aggregation/collection classes

Why Immutability?

• Simple – Exactly one state

– Potentially easier to design, implement, use, reason

about & make secure

• Inherently referentially transparent – Potential for optimisation

• Can be shared freely – Including “constant” aggregations of immutables

– Including persistent structures of immutables

– Suitable for caching

– Can even cache “pure” expressions involving immutables, e.g. 3 + 4, “string”.size(), fib(42)

– Inherently thread safe

Approaches to managing collection storage

• Mutable • Persistent

2006-2

• Immutable

‘c’ ‘a’ ‘c’ ‘a’

Add ‘t’ Add ‘t’ Add ‘t’

‘c’ ‘a’ ‘t’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’

‘c’

‘a’

‘t’

‘c’

‘a’

2006-2

• Immutable

‘c’ ‘a’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’

‘c’

‘a’

‘t’

‘c’

‘a’

Immutable practices

• Using mutating style

String invention = 'Mouse Trap' List inventions = [invention] invention = 'Better ' + invention inventions << invention assert inventions == ['Mouse Trap', 'Better Mouse Trap'] inventions.removeAll 'Mouse Trap' assert inventions == ['Better Mouse Trap']

Immutable practices

• Using mutating style

– We could possibly get away with this code here but it

has some debatable code smells

• (1) add a reference to a mutable list

• (2) change string reference losing original

• (3),(4) mutate list

• (4) duplicate first invention because original lost

String invention = 'Mouse Trap' List inventions = [invention] //(1) invention = 'Better ' + invention //(2) inventions << invention //(3) assert inventions == ['Mouse Trap', 'Better Mouse Trap'] inventions.removeAll 'Mouse Trap' //(4) assert inventions == ['Better Mouse Trap']

2006-2

• Immutable

‘c’ ‘a’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’

‘c’

‘a’

‘t’

‘c’

‘a’

Immutable practices

• Avoid using mutator methods

Avoid Prefer

list.sort() list.sort(false)

list.unique() list.unique(false)

list.reverse(true) list.reverse()

list.addAll list.plus

list.removeAll list.minus

String or List += or << use differently named variables

mutating java.util.Collections

void methods, e.g. shuffle, swap,

fill, copy, rotate

your own non mutating variants

Immutable practices

Avoid Prefer

fill, copy, rotate

public class Collections { public static void shuffle(List<?> list) { /* ... */ } /* ... */ }

Immutable practices

Avoid Prefer

fill, copy, rotate

static List myShuffle(List list) { List result = new ArrayList(list) Collections.shuffle(result) result }

Immutable practices

– But only marginal gains when using Java’s built-in

collections

// Avoid String invention = 'Mouse Trap' List inventions = [invention] invention = 'Better ' + invention inventions << invention assert inventions == ['Mouse Trap', 'Better Mouse Trap'] inventions.removeAll 'Mouse Trap' assert inventions == ['Better Mouse Trap'] // Prefer String firstInvention = 'Mouse Trap' List initialInventions = [firstInvention] String secondInvention = 'Better ' + firstInvention List allInventions = initialInventions + secondInvention assert allInventions == ['Mouse Trap', 'Better Mouse Trap'] List bestInventions = allInventions - firstInvention assert bestInventions == ['Better Mouse Trap']

Immutability options - collections

• Built-in

• Google Collections

– Numerous improved immutable collection types

• Groovy run-time metaprogramming

import com.google.common.collect.* List<String> animals = ImmutableList.of("cat", "dog", "horse") animals << 'fish' // => java.lang.UnsupportedOperationException

def animals = ['cat', 'dog', 'horse'].asImmutable() animals << 'fish' // => java.lang.UnsupportedOperationException

def animals = ['cat', 'dog', 'horse'] ArrayList.metaClass.leftShift = { throw new UnsupportedOperationException() } animals << 'fish' // => java.lang.UnsupportedOperationException

2006-2

• Immutable

‘c’ ‘a’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’ ‘c’ ‘a’

‘c’ ‘a’ ‘t’

‘c’

‘a’

‘t’

‘c’

‘a’

Immutability – persistent collections

• Functional Java – Or Functional Groovy, clj-ds, pcollections, totallylazy

@Grab('org.functionaljava:functionaljava:3.1') def pets = fj.data.List.list("cat", "dog", "horse") // buy a fish def newPets = pets.cons("fish") assert [3, 4] == [pets.length(), newPets.length()]

newPets

• Functional Java

@Grab('org.functionaljava:functionaljava:3.1') def pets = fj.data.List.list("cat", "dog", "horse") def newPets = pets.cons("fish") assert [3, 4] == [pets.length(), newPets.length()] // sell the horse def remaining = newPets.removeAll{ it == 'horse' }

newPets

tail head

remaining

• Functional Java

@Grab('org.functionaljava:functionaljava:3.1') def pets = fj.data.List.list("cat", "dog", "horse") def newPets = pets.cons("fish") assert [3, 4] == [pets.length(), newPets.length()] def remaining = newPets.removeAll{ it == 'horse' } assert [3, 4, 3] == [pets, newPets, remaining]*.length()

newPets

tail head

remaining

• Functional Java

@Grab('org.functionaljava:functionaljava:3.1') def pets = fj.data.List.list("cat", "dog", "horse") def newPets = pets.cons("fish") assert [3, 4] == [pets.length(), newPets.length()] def remaining = newPets.removeAll{ it == 'horse' } assert [3, 4, 3] == [pets, newPets, remaining]*.length()

newPets

tail head

remaining

D E F G

original

• You will see the correct results but in general, different

operations may give very differing performance

characteristics from what you expect – But don’t fret, smart people are working on smart structures to support a variety

of scenarios. You may even have several in your current NoSQL implementation

D E F G

original modified

Reality check

• OK, do I have to write this myself? – Might pay to try some simple ones. Take a look at Eric

Lippert’s blog on some C# implementations. Here is

the first part (Part 1: Kinds of Immutability):

http://blogs.msdn.com/ericlippert/archive/2007/11/13/

immutability-in-c-part-one-kinds-of-immutability.aspx

– Also consider

• Part 2: Simple Immutable Stack, Part 3: Covariant Immutable

Stack, Part 4: Immutable Queue, Part 6: Simple Binary Tree

– There are probably plenty of implementations you can

already use

– See also: Purely Functional Data Structures by Chris

Okasak, Cambridge University Press (1999)

• It turns out you use trees for nearly everything!

Reality check

• Functional Java persistent data structures – Singly-linked list (fj.data.List)

– Lazy singly-linked list (fj.data.Stream)

– Nonempty list (fj.data.NonEmptyList)

– Optional value (a container of length 0 or 1) (fj.data.Option)

– Immutable set using a red/black tree (fj.data.Set)

– Immutable multi-way tree (a.k.a. rose tree) (fj.data.Tree)

– Immutable tree-map using a red/black tree (fj.data.TreeMap)

– Products (tuples) of arity 1-8 (fj.P1..P8)

– Vectors of arity 2-8 (fj.data.vector.V2..V8)

– Pointed lists and trees (fj.data.Zipper and fj.data.TreeZipper)

– Type-safe, generic heterogeneous list (fj.data.hlist.HList)

– Immutable arrays (fj.data.Array)

– Disjoint union datatype (fj.data.Either)

– 2-3 finger trees supporting access to the ends in amortized O(1)

time (fj.data.fingertrees)

Reality check

• OK, have we achieved something simpler? – It depends. Understanding the insides of persistent

data structures can be very hard

• But as you move towards more complex systems and more

concurrent systems, not having to worry about which

threads are mutating what and when usually outweighs the

complexities of using persistent data structures

• Arguing for impure in Haskell:

http://www.cse.unsw.edu.au/~benl/papers/thesis/lippmeier-

impure-world.pdf

– They still don’t solve all of the problems. For any

significant problem you will have multiple threads

working on the solution. In some sense we have just

moved the problem but at least we have separated

concerns.

• You might combine with message passing (actors) or

dataflow or software transactional memory (STM)

Immutable Classes

• Some Rules – Don’t provide mutators

– Ensure that no methods can

be overridden

• Easiest to make the class final

• Or use static factories & non-public

constructors

– Make all fields final

– Make all fields private

• Avoid even public immutable constants

– Ensure exclusive access to any mutable components

• Don’t leak internal references

• Defensive copying in and out

– Optionally provide equals and hashCode methods

– Optionally provide toString method

@Immutable...

• Java Immutable Class – As per Joshua Bloch

Effective Java

2006-2

public final class Person { private final String first; private final String last; public String getFirst() { return first; } public String getLast() { return last; } @Override public int hashCode() { final int prime = 31; int result = 1; result = prime * result + ((first == null) ? 0 : first.hashCode()); result = prime * result + ((last == null) ? 0 : last.hashCode()); return result; } public Person(String first, String last) { this.first = first; this.last = last; } // ...

// ... @Override public boolean equals(Object obj) { if (this == obj) return true; if (obj == null) return false; if (getClass() != obj.getClass()) return false; Person other = (Person) obj; if (first == null) { if (other.first != null) return false; } else if (!first.equals(other.first)) return false; if (last == null) { if (other.last != null) return false; } else if (!last.equals(other.last)) return false; return true; } @Override public String toString() { return "Person(first:" + first + ", last:" + last + ")"; } }

...@Immutable...

• Java Immutable Class – As per Joshua Bloch

Effective Java

2006-2

public final class Person { private final String first; private final String last; public String getFirst() { return first; } public String getLast() { return last; } @Override public int hashCode() { final int prime = 31; int result = 1; result = prime * result + ((first == null) ? 0 : first.hashCode()); result = prime * result + ((last == null) ? 0 : last.hashCode()); return result; } public Person(String first, String last) { this.first = first; this.last = last; } // ...

// ... @Override public boolean equals(Object obj) { if (this == obj) return true; if (obj == null) return false; if (getClass() != obj.getClass()) return false; Person other = (Person) obj; if (first == null) { if (other.first != null) return false; } else if (!first.equals(other.first)) return false; if (last == null) { if (other.last != null) return false; } else if (!last.equals(other.last)) return false; return true; } @Override public String toString() { return "Person(first:" + first + ", last:" + last + ")"; } }

boilerplate

...@Immutable

2006-2

@Immutable class Person { String first, last }

Topics

Laziness & Strictness

• Type Safety

• More Info

2006-2

totallylazy library

• Similar to Groovy’s collection

GDK methods …

• Except … lazy …

@GrabResolver('http://repo.bodar.com/') @Grab('com.googlecode.totallylazy:totallylazy:1113') import static com.googlecode.totallylazy.Sequences.map import static com.googlecode.totallylazy.numbers.Numbers.* assert range(6, 10) == [6,7,8,9,10] assert range(6, 10, 2).forAll(even) assert range(6, 10).reduce{ a, b -> a + b } == 40 assert range(6, 10).foldLeft(0, add) == 40 assert map(range(6, 10), { it + 100 }) == [106,107,108,109,110] assert primes().take(10) == [2,3,5,7,11,13,17,19,23,29] assert range(1, 4).cycle().drop(2).take(8) == [3,4,1,2,3,4,1,2]

println range(6, 1_000_000_000_000).filter(even).drop(1).take(5) // => 8,10,12,14,16 (a handful of millis later)

Immutability options - collections

• This script

• Produces this output (order will vary)

@GrabResolver('http://repo.bodar.com/') @Grab('com.googlecode.totallylazy:totallylazy:1113') import static com.googlecode.totallylazy.Sequences.flatMapConcurrently import static com.googlecode.totallylazy.numbers.Numbers.* println flatMapConcurrently(range(6, 10)) { println it // just for logging even(it) ? [it, it+100] : [] }

//9 //7 //8 //6 //10 //6,106,8,108,10,110

GPars and TotallyLazy library

2006-2

@GrabResolver('http://repo.bodar.com') @Grab('com.googlecode.totallylazy:totallylazy:1113') import static groovyx.gpars.GParsExecutorsPool.withPool import static com.googlecode.totallylazy.Callables.asString import static com.googlecode.totallylazy.Sequences.sequence withPool { pool -> assert ['5', '6'] == sequence(4, 5, 6) .drop(1) .mapConcurrently(asString(), pool) .toList() }

withPool { assert ['5', '6'] == [4, 5, 6] .drop(1) .collectParallel{ it.toString() } }

<= Plain GPars equivalent

Groovy Streams

• https://github.com/timyates/groovy-stream

@Grab('com.bloidonia:groovy-stream:0.5.2') import groovy.stream.Stream // Repeat an object indefinitely Stream s = Stream.from { 1 } assert s.take( 5 ).collect() == [ 1, 1, 1, 1, 1 ] // Use an Iterable s = Stream.from 1..3 assert s.collect() == [ 1, 2, 3 ] // Use an iterator def iter = [ 1, 2, 3 ].iterator() s = Stream.from iter assert s.collect() == [ 1, 2, 3 ] // Use a map of iterables s = Stream.from x:1..2, y:3..4 assert s.collect() == [ [x:1,y:3],[x:1,y:4],[x:2,y:3],[x:2,y:4] ]

Groovy Streams

• https://github.com/dsrkoc/monadologie

import static hr.helix.monadologie.MonadComprehension.foreach def res = foreach { a = takeFrom { [1, 2, 3] } b = takeFrom { [4, 5] } yield { a + b } } assert res == [5, 6, 6, 7, 7, 8]

Functional Groovy

• https://github.com/mperry/functionalgroovy

@GrabResolver('https://oss.sonatype.org/content/groups/public') @Grab('com.github.mperry:functionalgroovy-core:0.2-SNAPSHOT') @Grab('org.functionaljava:functionaljava:3.1') import static com.github.mperry.fg.Comprehension.foreach 1.to(5).each { println it } def result = foreach { num { 1.to(2) } yield { num + 1 } } assert result.toJList() == [2, 3]

Show me the code

LazyMain.groovy

Topics

GPars & Concurrency

• Type Safety

• More Info

2006-2

Ralph Johnson: Parallel Programming

• Styles of parallel programming – Threads and locks

• Nondeterministic, low-level, rumored humans can do this

– Asynchronous messages e.g. Actors –

no or limited shared memory • Nondeterministic, ok for I/O but be careful with side-effects

– Sharing with deterministic restrictions

e.g. Fork-join • Hopefully deterministic semantics, not designed for I/O

– Data parallelism • Deterministic semantics, easy, efficient, not designed for I/O

2006-2

http://strangeloop2010.com/talk/presentation_file/14485/Johnson-DataParallelism.pdf

Each approach has some caveats

GPars • http://gpars.codehaus.org/

• Library classes and DSL sugar providing

intuitive ways for Groovy developers to

handle tasks concurrently. Logical parts:

– Data Parallelism features use JSR-166y Parallel Arrays

to enable multi-threaded collection processing

– Asynchronous functions extend the Java 1.5 built-in

support for executor services to enable multi-threaded

closure processing

– Dataflow Concurrency supports natural shared-memory

concurrency model, using single-assignment variables

– Actors provide an implementation of Erlang/Scala-like

actors including "remote" actors on other machines

– Safe Agents provide a non-blocking mt-safe reference to

mutable state; inspired by "agents" in Clojure

2006-2

Coordination approaches S

roovy in

Data Parallelism:

Fork/Join

Map/Reduce

Fixed coordination

(for collections)

Actors Explicit coordination

Safe Agents Delegated coordination

Dataflow Implicit coordination

GPars: Choosing approaches F

ttp://g

Parallel

Collections

Data Parallelism

Parallelism

Streamed Data

Parallelism

Dataflow

operators

Actors

Dataflow tasks

Actors

Asynch fun’s

Immutable

Stm, Agents

Special collections

Synchronization

Linear Recursive

Linear

Recursive

Shared

Irregular Regular

Groovy Sequential Collection

2006-2

def oneStarters = (1..30) .collect { it ** 2 } .findAll { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.max() == 196 assert oneStarters.sum() == 747

GPars Parallel Collections…

2006-2

import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30) .collectParallel { it ** 2 } .findAllParallel { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.maxParallel() == 196 assert oneStarters.sumParallel() == 747 }

…GPars Parallel Collections

• Suitable when – Each iteration is independent, i.e. not:

fact[index] = index * fact[index - 1]

– Iteration logic doesn’t use non-thread safe code

– Size and indexing of iteration are important

2006-2

import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30) .collectParallel { it ** 2 } .findAllParallel { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.maxParallel() == 196 assert oneStarters.sumParallel() == 747 }

Parallel Collection Variations

• Apply some Groovy metaprogramming

2006-2

import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30).makeConcurrent() .collect { it ** 2 } .findAll { it ==~ '1.*' } .findAll { it ==~ '...' } assert oneStarters == [100, 121, 144, 169, 196] }

import groovyx.gpars.ParallelEnhancer def nums = 1..5 ParallelEnhancer.enhanceInstance(nums) assert [1, 4, 9, 16, 25] == nums.collectParallel{ it * it }

GPars parallel methods for collections Transparent Transitive? Parallel Lazy?

any { ... } anyParallel { ... } yes

collect { ... } yes collectParallel { ... }

count(filter) countParallel(filter)

each { ... } eachParallel { ... }

eachWithIndex { ... } eachWithIndexParallel { ... }

every { ... } everyParallel { ... } yes

find { ... } findParallel { ... }

findAll { ... } yes findAllParallel { ... }

findAny { ... } findAnyParallel { ... }

fold { ... } foldParallel { ... }

fold(seed) { ... } foldParallel(seed) { ... }

grep(filter) yes grepParallel(filter)

groupBy { ... } groupByParallel { ... }

max { ... } maxParallel { ... }

max() maxParallel()

min { ... } minParallel { ... }

min() minParallel()

split { ... } yes splitParallel { ... }

sum sumParallel // foldParallel +

Transitive means result is automatically transparent; Lazy means fails fast

GPars: Map-Reduce

2006-2

import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30).parallel .map { it ** 2 } .filter { it ==~ '1.*' } assert oneStarters.collection == [1, 16, 100, 121, 144, 169, 196] // aggregations/reductions assert oneStarters.max() == 196 assert oneStarters.reduce { a, b -> a + b } == 747 assert oneStarters.sum() == 747 }

GPars parallel array methods

Method Return Type

combine(initValue) { ... } Map

filter { ... } Parallel array

collection Collection

groupBy { ... } Map

map { ... } Parallel array

max() T

max { ... } T

min() T

min { ... } T

reduce { ... } T

reduce(seed) { ... } T

size() int

sort { ... } Parallel array

sum() T

parallel // on a Collection Parallel array

Parallel Collections vs Map-Reduce

Fork Fork

Join Join

Reduce

Filter

Filter Map

Concurrency challenge…

• Suppose we have the following

calculation involving several functions:

• And we want to use our available cores …

2006-2

// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] def a = 5 def b = f1(a) def c = f2(a) def d = f3(c) def f = f4(b, d) assert f == 10

…Concurrency challenge…

• We can analyse the example’s task graph:

2006-2

// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] def a = 5 def b = f1(a) def c = f2(a) def d = f3(c) def f = f4(b, d) assert f == 10

…Concurrency challenge…

• Manually using asynchronous functions:

2006-2

// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import static groovyx.gpars.GParsPool.withPool withPool(2) { def a = 5 def futureB = f1.callAsync(a) def c = f2(a) def d = f3(c) def f = f4(futureB.get(), d) assert f == 10 }

…Concurrency challenge

• And with GPars Dataflows:

2006-2

def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task new Dataflows().with { task { a = 5 } task { b = f1(a) } task { c = f2(a) } task { d = f3(c) } task { f = f4(b, d) } assert f == 10 }

…Concurrency challenge

• And with GPars Dataflows:

2006-2

def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task new Dataflows().with { task { f = f4(b, d) } task { d = f3(c) } task { c = f2(a) } task { b = f1(a) } task { a = 5 } assert f == 10 }

GPars: Dataflows...

2006-2

import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.result = flow.x + flow.y } task { flow.x = 10 } task { flow.y = 5 } assert 15 == flow.result

new DataFlows().with { task { result = x * y } task { x = 10 } task { y = 5 } assert 50 == result }

...GPars: Dataflows...

• Evaluating:

2006-2

import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.a = 10 } task { flow.b = 5 } task { flow.x = flow.a - flow.b } task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } assert flow.result == 75

result = (a – b) * (a + b)

Question: what happens if I change the order of the task statements here?

• Naive attempt for loops

2006-2

import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task final flow = new Dataflows() [10, 20].each { thisA -> [4, 5].each { thisB -> task { flow.a = thisA } task { flow.b = thisB } task { flow.x = flow.a - flow.b } task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } println flow.result } } // => java.lang.IllegalStateException: A DataflowVariable can only be assigned once.

... task { flow.a = 10 } ... task { flow.a = 20 }

Don’t do this!

2006-2

import groovyx.gpars.dataflow.DataflowStream import static groovyx.gpars.dataflow.Dataflow.* final streamA = new DataflowStream() final streamB = new DataflowStream() final streamX = new DataflowStream() final streamY = new DataflowStream() final results = new DataflowStream() operator(inputs: [streamA, streamB], outputs: [streamX, streamY]) { a, b -> streamX << a - b; streamY << a + b } operator(inputs: [streamX, streamY], outputs: [results]) { x, y -> results << x * y } [[10, 20], [4, 5]].combinations().each{ thisA, thisB -> task { streamA << thisA } task { streamB << thisB } } 4.times { println results.val }

...GPars: Dataflows

• Suitable when: – Your algorithms can be expressed as mutually-

independent logical tasks

• Properties: – Inherently safe and robust (no race conditions or

livelocks)

– Amenable to static analysis

– Deadlocks “typically” become repeatable

– “Beautiful” (declarative) code

2006-2

import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task final flow = new Dataflows() task { flow.x = flow.y } task { flow.y = flow.x }

…GPars: Actors...

2006-2

import static groovyx.gpars.actor.Actors.* def votes = reactor { it.endsWith('y') ? "You voted for $it" : "Sorry, please try again" } println votes.sendAndWait('Groovy') println votes.sendAndWait('JRuby') println votes.sendAndWait('Go') def languages = ['Groovy', 'Dart', 'C++'] def booth = actor { languages.each{ votes << it } loop { languages.size().times { react { println it } } stop() } } booth.join(); votes.stop(); votes.join()

You voted for Groovy

You voted for JRuby

Sorry, please try again

You voted for Groovy

Sorry, please try again

Software Transactional Memory…

2006-2

@Grab('org.multiverse:multiverse-beta:0.7-RC-1') import org.multiverse.api.references.LongRef import static groovyx.gpars.stm.GParsStm.atomic import static org.multiverse.api.StmUtils.newLongRef class Account { private final LongRef balance Account(long initial) { balance = newLongRef(initial) } void setBalance(long newBalance) { if (newBalance < 0) throw new RuntimeException("not enough money") balance.set newBalance } long getBalance() { balance.get() } } // ...

…Software Transactional Memory

2006-2

// ... def from = new Account(20) def to = new Account(20) def amount = 10 def watcher = Thread.start { 15.times { atomic { println "from: ${from.balance}, to: ${to.balance}" } sleep 100 } } sleep 150 try { atomic { from.balance -= amount to.balance += amount sleep 500 } println 'transfer success' } catch(all) { println all.message } atomic { println "from: $from.balance, to: $to.balance" } watcher.join()

Topics

Type Safety

• More Info

2006-2

Show me the code

JScience, SPrintfChecker, GenericStackTest

Topics

• Type Safety

Word Split (bonus material)

• More Info

2006-2

Word Split with Fortress

2006-2

Guy Steele’s StrangeLoop keynote (from slide 52 onwards for several slides):

http://strangeloop2010.com/talk/presentation_file/14299/GuySteele-parallel.pdf

Word Split…

2006-2

def swords = { s -> def result = [] def word = '' s.each{ ch -> if (ch == ' ') { if (word) result += word word = '' } else word += ch } if (word) result += word result }

assert swords("This is a sample") == ['This', 'is', 'a', 'sample'] assert swords("Here is a sesquipedalian string of words") == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words']

Word Split…

2006-2

Word Split…

2006-2

…Word Split…

2006-2

…Word Split…

2006-2

Segment(left1, m1, right1) Segment(left2, m2, right2)

Segment(left1, m1 + [ ? ] + m2, right2)

…Word Split…

2006-2

…Word Split…

2006-2

class Util { static maybeWord(s) { s ? [s] : [] } } import static Util.* @Immutable class Chunk { String s public static final ZERO = new Chunk('') def plus(Chunk other) { new Chunk(s + other.s) } def plus(Segment other) { new Segment(s + other.l, other.m, other.r) } def flatten() { maybeWord(s) } } @Immutable class Segment { String l; List m; String r public static final ZERO = new Segment('', [], '') def plus(Chunk other) { new Segment(l, m, r + other.s) } def plus(Segment other) { new Segment(l, m + maybeWord(r + other.l) + other.m, other.r) } def flatten() { maybeWord(l) + m + maybeWord(r) } }

…Word Split…

2006-2

def processChar(ch) { ch == ' ' ? new Segment('', [], '') : new Chunk(ch) }

def swords(s) { s.inject(Chunk.ZERO) { result, ch -> result + processChar(ch) } }

assert swords("Here is a sesquipedalian string of words").flatten() == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words']

…Word Split…

2006-2

…Word Split…

2006-2

…Word Split…

2006-2

THREADS = 4 def pwords(s) { int n = (s.size() + THREADS - 1) / THREADS def map = new ConcurrentHashMap() (0..<THREADS).collect { i -> Thread.start { def (min, max) = [ [s.size(), i * n].min(), [s.size(), (i + 1) * n].min() ] map[i] = swords(s[min..<max]) } }*.join() (0..<THREADS).collect { i -> map[i] }.sum().flatten() }

…Word Split…

2006-2

import static groovyx.gpars.GParsPool.withPool THRESHHOLD = 10 def partition(piece) { piece.size() <= THRESHHOLD ? piece : [piece[0..<THRESHHOLD]] + partition(piece.substring(THRESHHOLD)) } def pwords = { input -> withPool(THREADS) { partition(input).parallel.map(swords).reduce{ a, b -> a + b }.flatten() } }

…Guy Steele example in Groovy…

2006-2

def words = { s -> int n = (s.size() + THREADS - 1) / THREADS def min = (0..<THREADS).collectEntries{ [it, [s.size(),it*n].min()] } def max = (0..<THREADS).collectEntries{ [it, [s.size(),(it+1)*n].min()] } def result = new DataFlows().with { task { a = swords(s[min[0]..<max[0]]) } task { b = swords(s[min[1]..<max[1]]) } task { c = swords(s[min[2]..<max[2]]) } task { d = swords(s[min[3]..<max[3]]) } task { sum1 = a + b } task { sum2 = c + d } task { sum = sum1 + sum2 } println 'Tasks ahoy!' sum } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } }

DataFlow version: partially hard-coded to 4 partitions for easier reading

…Guy Steele example in Groovy…

2006-2

GRANULARITY_THRESHHOLD = 10 THREADS = 4 println GParsPool.withPool(THREADS) { def result = runForkJoin(0, input.size(), input){ first, last, s -> def size = last - first if (size <= GRANULARITY_THRESHHOLD) { swords(s[first..<last]) } else { // divide and conquer def mid = first + ((last - first) >> 1) forkOffChild(first, mid, s) forkOffChild(mid, last, s) childrenResults.sum() } } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } }

Fork/Join version

…Guy Steele example in Groovy

2006-2

println GParsPool.withPool(THREADS) { def ans = input.collectParallel{ processChar(it) }.sum() switch(ans) { case Chunk: return maybeWord(ans.s) case Segment: return ans.with{ maybeWord(l) + m + maybeWord(r) } } }

Just leveraging the algorithm’s parallel nature

Topics

• Type Safety

More Info

2006-2

More Information: Groovy in Action

functional groovy

Technology

groovy tutorial

groovy grails20110120présentation sur groovy / grails par...

groovy tutorial -...

groovy scripts with groovy

groovy adf

groovy shapes

programming groovy 2 -...

groovy basics

programming groovy...

functional groovy

groovy vampires: combining groovy, rest, nosql, and more

discovering functional treasure in idiomatic groovy

restful groovy

groovy 2.5+ roadmap - objectcomputing.com · • home of...

functional programming matters - accu.org€¦ · java,...

adventures in groovy – part 20: groovy on-premise vs

groovy cotbed bed groovy lit groovy · groovy groovy bed...

groovy 2.5 features & groovy 3+ roadmap...what is groovy?...

functional programming in groovy

groovy and grails - chariot...