Functional programming and new teams

Michael Neale@michaelneale

developer.cloudbees.com

Background

Work @cloudbees: developer.cloudbees.comFP on and off since uni (first programming language was miranda)!github.com/michaelneale!polyglot - but mostly cussing at bash scripts nowadays **!

Context

New CompanyNew TeamNew Product!You might not be this “lucky”But I hope you get some ideas or inspiration!

History

2010 Started: JVM stack parts - Scala not controversial (I had experience) - working mostly “lone wolf”!2011 - another team added - brought Erlang!

Erlang!Me

other team members

More FP

Given “success” with Scala, another “FP language” was not a great risk. !

Since then

More developers hired!Existing developers start to care about FP!I learned Erlang (avoiding the one person wolfpack)!Clojure & “micro services” introduced

Observation

FP means one person can do more && People like me like to work alone!∴ risk of staying with one-person-per app(aside: is this a bad thing?)

Maintainability objectionIt goes: - how will anyone be able to maintain this after you?!My Experience: - 2 projects featuring FP handed over successfully - new developers able to pick up FP easily - seasoned developers too!My anecdotes are clearly science!!

Where we are today

Several systems involving: - Erlang (on every server and core services) - Scala (cloud controllers) - Clojure (micro services - talk to github api)

Where we may differ

Small teams - many systems.!∴ Little overlap in jobs.

What did we do with FPManage horrendous public cloud APIs!Server controlling (agent) - manage horrendously misbehaving apps!Automatic scaling!Github crawling!

Surprisingly practical things

No real calculations, no explicit maths.!Just boring every day error prone stuff.

For example (provisioning)

Providore Evil cloud api

Build masters

build workers Workspace storage

Every step involves failures, retries!Every step requires unknown wait times.

The problem:

How can we solve this?TDD? problems only manifest under load/in-production. APIs are buggy, change over time.!Industry best practice: scripting: 60% of the time, it works every time.!How does FP help?

YesTypes (providore is written in scala) - Specifically: Maybe/Option, Either - Closed Data Types (servers only in so many states)!The M word: Monads!Currying

Cloud API

launch_server: Server!at best hopes to be: !launch_server: Maybe[Server]launch_server: Either[Server, OhGodWhyWhyWhy]!(not an actual pure function of course)!

Cloud Monad

Cloud APIs are like IO!Slow, horrible, misbehaving IO!...and then the APIs other people write!All want to be monadic

val validation: String \/ Subscription = (for {! account <- extractAccount! _ <- validateSubscription(account)! callback <- extractCallBack! plan <- validatePlan ! billing_day <- extractBillingDay! subscription <- createSub(account, plan, callback, ...! } yield subscription).run(req.body)!

(scala) ReaderT to help you compose !

http://debasishg.blogspot.com.au/2011/07/monad-transformers-in-scala.html

!

Need to “organise code in monadic way”

!!!Infrastructure devs, “devops”: pay attention!!Correct code matters here: hard to test -> correctness can help!We can do better

TypesSo hard to catch things without them!Monadic IO + types mean you catch things before you try!Trying/experimenting can be $$ expensive...!!!!

Types helped with

Ignored messagesBad pattern matchingMisconfiguration/timing of server creationAvoiding “stringly typed” messages!

Types didn’t help with...

! !zombies = ec2.servers.select { |i| i.tags.empty? && tenured?(i) && alive?(i) }!Parallel.each(zombies, :in_threads => 15) do |zombie|! begin! puts "Terminating zombie node #{zombie.id}"! ec2.servers.get(zombie.id).destroy! end!end!

CurryingServer lifecycle: Reserved->Launching->Update (user data)->Volume Create->Volume Attach->initialise/start!Accumulate setup data via partial application!Instead of an object that has mutating state, a function you partially apply to accumulate data.!Good “beginner” FP concept (powerful, simple)

Another example (autoscale)

Message Bus Autoscale

Controller

Servers/

Auto scalingF(last minute stats, previous data window) -> “Suggestion” to scale up, or down (or in or out)!Fundamentally calculation, fundamentally functional.!Built in Erlang.

Side effects

Push out side effects to other side of the message bus!Let a nasty app handle the side effects!Messages are signals, suggestions, idempotent!!

Developers Developers

Don’t “sell” to your developers by: - saying monad too often - saying “it’s easy” - showing that it can be just as easy/familiar as what they have!Instead...

Find the functionsFind the functions in what they doFind the calculations (eg core of autoscaling)!Intermediate: Currying, Higher order functions, Types (good ones)!Advanced:Monadic

Unlearning OO

Erlang and Clojure: both excellent at teaching people to forget about OO. !Scala: challenge. Temptation always there. ∴ Use object/package as namespace/modules!

Advocating FP ...

Ruby Slippers

Concept from _why!You can’t wear anything you want, but you can wear ruby slippers!http://viewsourcecode.org/why/hacking/wearingRubySlippersToWork.html!Use ruby for small, but necessary, tools/scripts!

Introducing ...

Lambda Underpants

Lambda Underpantshttp://lambdaunderpants.com!Use FP for small, but necessary services, scripts!Disposable, possibly forgettable, sneaky. !example: Kit by @nkpartme: small clojure services that deal with github api

Micro ServicesSmall http services (typically) - REST!perfect for FP: F(request) -> response!Examples: - github crawling - monitoring cloud usage (instances) - admin interfaces and utilities

Micro ServicesPick ones that are “transforming” data from one service to a client!Low risk, uncontroversial!Ask forgiveness later.

ALT:Mixed language projects

Good idea??Relevant to JVM (and .net) environs only(?)!Ease-into-itJury is out...

Final ObservationDevelopers who have fondness for emacs/vim (over IDE) find things easier!FP invites “change this small bit, see what happens” exploration!

Thank you

Michael Nealehttps://twitter.com/michaelnealehttps://developer.cloudbees.com

Adop%ng  

Func%onal  Programming  at  IOOF  

 -­‐  an  experimental  approach  

Kornelis  Sietsma  -­‐  @kornys  

The  Project  

!"#$

%&'()*+&,

!"#$ !"#$

-"./&0!"#$

-"./&0!"#$

-"./&0!"#$

-"./&0!"#$

Constraints  ...DecNovOctSepAugJulJunMayAprFebAug Oct MarSep JanDecNov

Phase&1

Phase&2

Deadline

3

4

and&on...

Deadline

Deadline

Phase  One  private final GuavaHelpers.ReduceFunction<List<Pair<TransactFile,! Transformer.Payload>>, TransactFile> fileTransformer! = new GuavaHelpers.ReduceFunction<List<Pair<TransactFile,! Transformer.Payload>>, TransactFile>() {! @Override! public List<Pair<TransactFile, Transformer.Payload>> ! apply(List<Pair<TransactFile, Transformer.Payload>> memo,! TransactFile file) {! Transformer.Payload transformedContent                      = findTransformer(file.getTransactionType())! .transform(file.getContents(),! file.getCreationTime().toDate());! memo.add(! new ImmutablePair<TransactFile, Transformer.Payload>! (file, transformedContent));! return memo;! }!};  

Why  not  FP?  

(defn file-­‐transform! [memo, file]! (let [transformer (find-transformer (:type file))! new-contents! (transform (:contents file) (to-date (:creation file)))]! (conj memo [file new-contents])))  

“But…”  

!"#$"$%%

&'()$

*+%%+"#%+,-$.

/'%0$12

“But…”  

Why  not  try  it?  

“Concurrent  set-­‐based  engineering”  

“OK  -­‐  You  have  three  weeks”  

...DecNovOctSepAugJulJunMayAprFebAug Oct MarSep JanDecNov

Phase&1

Phase&2

Deadline

3

4

and&on...

Deadline

Deadline

!

The  Plan  

!"#$

%&'()*+&,

!"#$ !"#$

-"./&0!"#$

-"./&0!"#$

-"./&0!"#$

-"./&0!"#$

Evalua%on  Criteria  

•  Testability  •  Speed  of  development  •  Ease  of  change  •  Ease  of  maintenance  •  Learning  curve  •  Community  and  support  •  Libraries  and  tools  

Methodology  

Methodology  

Methodology  

Methodology  

Methodology  

Metrics  -­‐  Objec%ve  

•  Speed  of  development  •  Time  use  breakdown  – Wri%ng  code  – Wri%ng  tests  – Dealing  with  tooling  – Adding  libraries  

•  Quality  of  tools  and  libraries  

Metrics  –  Subjec%ve  

•  Coding  •  Tes%ng  •  Problem  solving  •  Changing  requirements  •  Tools  and  libraries  •  Overall  “gut  feel”  

A  note  on  context  

“It  Depends…”  

The  first  two  weeks  (defn –main  []! (println "hello  world"))  

object HelloWorld {! def main(args: Array[String]) {! println("Hello,  world!")! }!}  

public class HelloWorld {! public static void main(String[] args) {! System.out.println("Hello,  World");! }!}  

Objec%ve  %me  usage  

0  

500  

1000  

1500  

2000  

2500  

3000  

1   2   3   4   5   6   7   8  

Scala  

tools  

libs  

test  

code  

0  

500  

1000  

1500  

2000  

2500  

3000  

1   2   3   4   5   6   7   8  

Clojure  

tools  

libs  

test  

code  

0  

500  

1000  

1500  

2000  

2500  

1   2   3   4   5   6   7   8  

Java  

tools  

libs  

test  

code  

Subjec%ve  -­‐  Clojure  Overall

tests

code

tools/libs

!2#

!1#

0#

1#

2#

0# 1# 2# 3# 4# 5# 6# 7# 8# 9#

Subjec%ve  -­‐  Scala  Overall

tests

code

tools/libs

!2#

!1#

0#

1#

2#

0# 1# 2# 3# 4# 5# 6# 7# 8# 9#

Subjec%ve  -­‐  Java  Overall

tests

code

tools/libs

!2#

!1#

0#

1#

2#

0# 1# 2# 3# 4# 5# 6# 7# 8# 9#

A_er  two  weeks  

!"#$%

!2#

!1#

0#

1#

2#

0# 1# 2# 3# 4# 5# 6# 7# 8# 9#Java

Scala

Clojure

Clojure  technical  feasibility  

•  Database  access  •  XML  read/write  •  REST  API  •  SOAP  API  •  Handling  large  data  sets  •  Message  queues  •  Task  scheduling  •  Sta%c/Dynamic  web  •  …  and  many  more  

Clojure  technical  feasibility  

•  Database  access  •  XML  read/write  •  REST  API  •  SOAP  API  •  Handling  large  data  sets  •  Message  queues  •  Task  scheduling  •  Sta%c/Dynamic  web  •  …  and  many  more  

Yes  Yes  Yes  Via  Java  Yes  Yes  Yes  Yes  Yes,  yes,  yes  

So  that’s  it  –  Clojure  FTW?  

Experiment  review  Criteria   Subjec6ve  Ra6ngs  

Project    

Long  term  

 

Total     Bad   Poor   Ok   Good   Awesome  

Testability  -­‐  Unit   5   5   10   0   1   3   2   1  

Speed  of  development   8   1   9   1   1   1   2   2  

Testability  –  int/accept.   2   7   9   1   1   2   3   0  

Ease  of  change   3   5   8   1   2   2   2   0  

Ease  of  Maintenance   0   7   7   1   4   2   0   0  

Learning  Curve   2   4   6   1   2   2   1   1  

Community  /  Support   1   5   6   1   2   0   1   2  

Change  is  hard.  

So,  what  was  the  decision?  

“No.”  

So,  what  was  the  decision?  

“No.”  “Yes!”  

So  we  built  it  in  Clojure!  

Stuff Stuff Stuff

Stuff%System%B

Stuff%System%A

OtherSystems

What  about  those  deadlines?  

...DecNovOctSepAugJulJunMayAprFebAug Oct MarSep JanDecNov

Phase&1

Phase&2

Early!

3 4

Enhance

Early! Early!

and&on...

!

Lines  of  code  

0  

1000  

2000  

3000  

4000  

5000  

6000  

7000  

9/01/13   9/02/13   9/03/13   9/04/13   9/05/13   9/06/13   9/07/13   9/08/13   9/09/13   9/10/13  

lines  of  cod

e  

Clojure  code  

Clojure  test  

Java  code  

Java  test  

Javascript  code  

Javascript  test  

Clojure  ms  code  

Clojure  ms  test  

Lines  of  code  

0  

20  

40  

60  

80  

100  

120  

0  

1000  

2000  

3000  

4000  

5000  

6000  

7000  

9/01/13   9/02/13   9/03/13   9/04/13   9/05/13   9/06/13   9/07/13   9/08/13   9/09/13   9/10/13  

Busine

ss  Storie

s  (black  lin

e)  

lines  of  cod

e  

Clojure  code  

Clojure  test  

Java  code  

Java  test  

Javascript  code  

Javascript  test  

Clojure  ms  code  

Clojure  ms  test  

business  stories  

New  hires  

And  awesome  code.  (defn-­‐ dec-­‐first-­‐digit [digits]! (conj (rest digits) (dec (first digits))))!!(defn-­‐ apply-­‐weighting [weights values]! (map * values weights))!!(defn-­‐ abn [value]! (let [weighted-sum! (-­‐>> (map #(Integer/parseInt (str %)) value)! (dec-first-digit)! (apply-weighting [10 1 3 5 7 9 11 13 15 17 19])! (apply +))]! (if-­‐not (= (rem weighted-sum 89) 0) "Not  a  valid  ABN")))!!(defn-­‐ tfn [value]! (let [weighted-sum! (-­‐>> (map #(Integer/parseInt (str %)) value)! (apply-weighting (if (= (count value) 8)! [10 7 8 4 6 3 5 1]! [10 7 8 4 6 3 5 2 1]))! (apply +))]! (if-­‐not (= (rem weighted-sum 11) 0) "Not  a  valid  TFN")))  

Conclusions  

•  Experiments  are  awesome  •  Change  is  hard  •  Metrics  are  hard  •  Plan  ahead  •  Remember,  it’s  all  about  your  own  context.  

Thank  you!  

Kornelis  Sietsma  -­‐  @kornys  

Credits:  “classic  text  editor  learning  curve”  –  pi@nightstar.net  (apparently)  “distraught  func%onal  programmer”  -­‐  @oorom    

adopting fp

who am I?

2.04Nebuchadnezzar

Jed Wesley-Smith

@jedws

concurrency performance bass player

7.13The Tardis

what is fp?

purethey don’t change anything

totalthey accept all values in the domain, no exceptions!

referentially transparenta call can be replaced by its result without changing the meaning of the program

we can change the number of times something is called and the order in which they are called without affecting the result of the program – fearless refactoring!

programming with functions

immutablethey don’t change

comparablecan be considered equal, or ordered against each other;cannot compare things that change

sharablethread-safe, don’t need to worry about locks, or ownership, or resource management

functions are values too!

programming with values

Rich Hickey, Are We There Yet JVM Language Summit 2009 Keynote

“we invented mutable values, we must uninvent them.”

Tony Morris adapted from @fogus

“we must immutilate them.”

why fp?

2.09Rapture

simple

functional programming is fundamentally simple

programming is hard

concurrency is really hard

a simple a model as possible makes it easier to reason about our programs

reliable

programs that are reasonable

where invariants cannot be arbitrarily violated by non-deterministic ordering of changes to the state of the running program

yield to logical proofs that certain pathological states cannot happen!

3.04The Derek Zoolander Centre for

Kids Who Can’t Read Goodand Wanna Learn to Do Other Stuff Good Too

how did we get there?

bugs

large old Java code-base, lots of accidental sharing of JavaBeans. Strange, unrepeatable, customer support cases

assigned engineer (often eventually me) would need to spend a large amount of time analysing the code,bending it, testing it and eventually coming up with a theory as to why it broke and how to fix it

surely, there’s a better way…

performance

when all your objects are mutable, you need to protect against accidental or malicious change to the references you pass out, locks and defensive copying are common

what is the cost of:

project.getAvatar().getAvatarId()

is it just a dereference? maybe a chain of dereferencing?

maybe it calls to the database? to a remote system?

how would you know?

immutability to the rescue

for many cases, simply making sure all objects that are passed in between classes and stored in caches are immutable is a big win

less copying

no thread-safety issues

no need for locks (lock contention can destroy performance in nasty ways)

big wins!we have used these very simple and basic techniques effectivelyto tame complexity in our products and libraries

\o/

6.01Avengers Mansion

but, there’s more…

7.07Discworld

2.01Black Mesa

digging into the fp toolkit

higher-order functions

higher-order functions take functions as inputs

Java (yet) doesn’t have language support for functions service.post(payload).map(new Function<Either<Failure, Key>, String>() { public String apply(Either<Failure, Key> input) { return input.fold(new Function<Failure, String>() { @Override public String apply(Failure fail) { log.error("Could not write. Error was {}", fail); return fail.toString(); } }, new Function<Key, String>() { public String apply(Key key) { log.warn("Successfully wrote to {}. Key is {}", BASE_LOCATION, key); return key.toString(); } }); } }).claim();

use a language that supports functions

for instance Scala service.put(payload).map { _.fold( fail => { log.error("Could not write. Error was {}", fail) fail.toString }, key => { log.warn("Successfully wrote to {}. Key is {}", BASE_LOCATION, key) key.toString } ) }.claim

what about functional architectures?

extend the idea to all parts of the system

don’t mutate anything

event sourcing, CQRS

git

lucene

datomic

case studies

FRAK7.11Galactica

marketplace.atlassian.com

business-critical application

80k LOC Java/Groovy/grails app

rife with performance & reliability issues

rewrite… with enormous time-pressure

started “traditional” service layer, DI etc.

but… prefer pure FP from start

only 2 in team with any Scala or FP experience

“hardcore” fp

Dependency Injection => functions with Reader monad

immutable domain model

updates with Lenses and State monad

straight-forward philosophy:push purity and type-safety to every corner of the system

result

a far more reliable system

4x faster than the original site

concise 4x functionality and 1/2 the code of the Grails app

and a pleasure to develop on, once you know what a Kleisli is

data analysis platform

data storage is content addressable

meta-data storage in datomic

architecture is pure FP

APIs expressed as in a monadic style, and executed using interpreters that run either in memory for testing or inside IO for file-system/network access

6.06Isengard

challenges

overcoming fear

unfamiliarity is deeply discomforting

fear and discomfort erode rationality, it is hard to rationalise about things that you don’t understand

some advanced functional programming topics such as category theory (monads, functors, applicatives) sound really scary

they aren’t really, they are just very general tools

education

continuous improvement read, study and practice, practice, practice!

you must unlearn, what you have learned shifting from abstraction over domain to abstraction over concepts takes a while to really appreciate

learning curve (especially with regard to IO)takes a while for benefits to materialise

invest wiselyfocus on the people who are interested, don’t spend too much time trying to win over the nay-sayers

teachingfind mentorsteams get great leg up from importing experience

allow innovation to distillaggressive and hard deadlines demand conservative approaches, and are fundamentally anti-agile

test everythingfunctional architectures lend themselves to testing, investigate and use property-based testing, avoid mock frameworks.

lessons

don’t do a bit functional you lose the benefits at a high level, with additional integration pain

compound effect of (in-)correctnesssmall hacks in the foundation can have large effects later on

embrace fearyou will be bewildered, and it is ok, it is only temporary

thanks!

images Sam Thebridge & Henry Tapia

The Empire doesn’tunderstand how security is supposedto work

7.09Death Star