@sixty_north

Understanding TransducersThrough Python

1

Austin Bingham@austin_bingham

Sunday, January 25, 15

237

“Transducers are coming”Rich Hickeyhttp://blog.cognitect.com/blog/2014/8/6/transducers-are-coming

Photo: Howard Lewis Ship under CC-BY

Sunday, January 25, 15

Functions which transform reducersWhat is a transducer?

‣Reducer (reducing function)• Any function we can pass to reduce(reducer, iterable[, initial])

• (result, input) → result

• add(result, input)reduce(add, [1, 2, 3], 0) → 6

‣Transducer (transform reducer)• A function which accepts a reducer, and transforms it in some way, and returns a

new reducer

• ((result, input) → result) → ((result, input) → result)3

Sunday, January 25, 15

Transducers are a functional programming technique not restricted to ClojureHow does this relate to Python?

‣Clojure implementation is the prototype/archetype• Heavily uses anonymous functions

• Uses overloads on function arity for disparate purposes

• Complected with existing approaches

‣Python implementation is pedagogical (but also useful)• Explicit is better than implicit

• Readability counts!

• Has all the functional tools we need for transducers

• I’m a better Pythonista than Clojurist4

Sunday, January 25, 15

5

Reviewfunctional programming

tools in

Sunday, January 25, 15

6

my_filter(is_prime, my_map(prime_factors, range(32) ) )

iterablesequence

sequence

Sunday, January 25, 15

7

def my_map(transform, iterable): def map_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, [])

def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, [])

Sunday, January 25, 15

7

def my_map(transform, iterable): def map_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, [])

def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, [])

reduce()

Sunday, January 25, 15

7

def my_map(transform, iterable): def map_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, [])

def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, [])

reduce()

sequence.append()

Sunday, January 25, 15

7

def my_map(transform, iterable): def map_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, [])

def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, [])

reduce()

sequence.append()

Empty list : ‘seed’must be a mutable

sequence

Sunday, January 25, 15

8

>>> reduce(make_mapper(square), range(10), []) [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> reduce(make_filterer(is_prime), range(100), []) [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

‣make_mapper() and make_filterer() are not composable

‣ to square primes we would need to call reduce() twice

‣ this requires we store the intermediate sequence

‣ the reducers returned by make_mapper() and make_filterer() depend on the seed being a mutable sequence e.g. a list

Sunday, January 25, 15

9

(defn map ([f] (fn [rf] (fn ([] (rf)) ([result] (rf result)) ([result input] (rf result (f input))) ([result input & inputs] (rf result (apply f input inputs))))))

Sunday, January 25, 15

10

(defn map ;; The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Forward to the zero-arity reducing function 'rf' ([result] (rf result)) ;; 1-arity : Forward to the one-arity reducing function 'rf' ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs))))))

Sunday, January 25, 15

11

(defn map ;; The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf' ([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs))))))

Sunday, January 25, 15

12

(defn map ;; The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf' ([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs))))))

To fully implement Clojure’s transducers in Python we also need:

‣ explicit association of the seed value with the reduction operation

‣ support for early termination without reducing the whole series

‣ reduction to a final value and opportunity to clean up state

Sunday, January 25, 15

13

class Reducer:

def __init__(self, reducer): # Construct from reducing function pass

def initial(self): # Return the initial seed value pass # 0-arity

def step(self, result, item): # Next step in the reduction pass # 2-arity

def complete(self, result): # Produce a final result and clean up pass # 1-arity

Sunday, January 25, 15

13

class Reducer:

def __init__(self, reducer): # Construct from reducing function pass

def initial(self): # Return the initial seed value pass # 0-arity

def step(self, result, item): # Next step in the reduction pass # 2-arity

def complete(self, result): # Produce a final result and clean up pass # 1-arity

new_reducer = Reducer(reducer)

Sunday, January 25, 15

13

class Reducer:

def __init__(self, reducer): # Construct from reducing function pass

def initial(self): # Return the initial seed value pass # 0-arity

def step(self, result, item): # Next step in the reduction pass # 2-arity

def complete(self, result): # Produce a final result and clean up pass # 1-arity

new_reducer = Reducer(reducer)

def transducer(reducer): return Reducer(reducer)

Sunday, January 25, 15

13

class Reducer:

def __init__(self, reducer): # Construct from reducing function pass

def initial(self): # Return the initial seed value pass # 0-arity

def step(self, result, item): # Next step in the reduction pass # 2-arity

def complete(self, result): # Produce a final result and clean up pass # 1-arity

new_reducer = Reducer(reducer)

def transducer(reducer): return Reducer(reducer) ⇐ two names for two concepts

Sunday, January 25, 15

Python Transducer implementations

14

Cognitecthttps://github.com/cognitect-labs

PyPI: transducers

• “official”

• Python in the style of Clojure

• Only eager iterables (step back from regular Python)

• Undesirable Python practices such as hiding built-ins

• Also Clojure, Javascript, Ruby, etc.

Sixty Northhttp://code.sixty-north.com/python-transducers

PyPI: transducer

• Pythonic

• Eager iterables

• Lazy iterables

• Co-routine based ‘push’ events

• Pull-requests welcome!

Sunday, January 25, 15

15

Thank you!

@sixty_north

Austin Bingham@austin_bingham

http://sixty-north.com/blog/

Sunday, January 25, 15