2013  overview Sagie Davidovich

singularityworld.com linkedin.com/in/sagied

2013  JVM  Languages  Landscape Static

Dynamic

Functional Object  O

riented

Who’s  using  scala

Performance  study  by  Google

Exponential  growth  in  demand

Thoughtworks  Technology  Radar  2012

Evaluate

Consider

Hold

Language  Design  Trade-­‐‑offs

Complexity

Expressiveness Performance

Positioning

JVM

Functional

Statically  Typed

Object  Oriented

Prof.  Martin  Odersky •  Designer of Java Generics •  Creator of JavaC •  Prof. at EPFL •  ACM Fellow •  Founder of

Class  Employee  -­‐‑  Java public class Employee implements Serializable { private final String firstName; private final String lastName;   public Employee (String firstName, String lastName) { this.firstName = firstName; this.lastName = lastName; }   public String getFirstName() { return lastName; }   public String getLastName() { return firstName; }   public Employee withFirstName(String firstName) { return new Employee (firstName, lastName); }   public Employee withLastName(String lastName) { return new Employee (firstName, lastName); }  

public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Employee employee = (Employee) o; if (firstName != null ? !firstName.equals(employee.firstName) : person.firstName != null) {

return false; } if (lastName != null ? !lastName.equals(employee.lastName) : employee.lastName != null) { return true; } return true; }   public int hashCode() { int result = firstName != null ? firstName.hashCode() : 0; result = 31 * result + (lastName != null ? lastName.hashCode() : 0); return result; }   public String toString() { return "Employee(" + firstName + "," + lastName + ")"; } }

Oops,  should  be  false  Anyone  noticed?

Class  Employee  -­‐‑  Scala

case class Employee(firstName: String, lastName: String)!

•  Constructor •  Copy  constructors •  Fields •  GeVers  /  (seVers) •  equals •  hashCode •  toString •  Recursive  decomposition •  Companion  object

Singletons Java public class OneAndOnly { private final static OneAndOnly INSTANCE = new OneAndOnly(); private OneAndOnly() {} public static OneAndOnly getInstance() { return OneAndOnly.INSTANCE; } }

Scala

object OneAndOnly

Everything  is  an  object

1.+(2) Same as

1 + 2

Every  block  returns  a  value  (no  statements,  only  expressions)

def add (x:Int, y:Int) = x + {val tmp = y; tmp * 3}!!if (x > 2) 3 else 4!!val doubled = for(i <- (1 to 10)) yield i * 2!

Type  inference scala> val x = 3 x: Int = 3

scala> def x(y:Int) = y * 9 x: (y: Int)Int

scala> def x(y:Int) = if (y == 1) List(y) else Set(y) x: (y: Int) Iterable[Int]

Higher  order  functions List(1, 2, 3).map((x: Int) => x + 1)

=

List(1, 2, 3).map(x => x + 1) // type is inferred

=

List(1, 2, 3).map(_ + 1) // placeholder notation

Universal  access  principle   import System._!!class Clz {! def w = currentTimeMillis / 1000! val x = currentTimeMillis / 1000! var y = currentTimeMillis / 1000! lazy val z = currentTimeMillis / 1000!}!

Default  methods  apply  and  unapply

object Square{ def apply(d: Double) = d * d def unapply(d: Double) = Some(math.sqrt(d)) } //apply val s = Square(3) // 9 // unaply 16 match { case Square(x) => x } // 4

Curried  functions •  Add new flow control and language constructs!

def loop(n: Int)(body: => Any) {! (0 until n) foreach (n => body)!}!!

loop(2) {! println("IM IN YR LOOP!")!}!

•  Multiple varargs lists are possible!!def crossProduct[L,R](left: L*)(right: R*) = for (l <- left; r <- right) yield (l,r)!crossProduct(1,2,3)(‘a’,’b’,’c’)!

•  Partially applied functions!!!

Multiple  inheritance  without  the  diamond  problem

Type  system

Strong

Static  +  Dynamic

Nominal  +  Structural

Explicit  +  Inferred

Erasure

Co/Contra-­‐‑variant

Value  +  Reference

Nonnullable Monads

Case  classes Path  dependent

Anonymous

Self  types

Type  aliases

Functional

Existential Implicit

Traits

Type  system

Type  safety  (Java  example) // compiles fine in Java (arrays are co-variant!!)

int[] ints = {3}; Object[] objects = ints; // ArrayStoreException in runtime

objects[0] = new Object();

Structural  types object Closer { def using(closeable: { def close(): Unit }, f: => Unit) { try { f } finally { closeable.close } } }

Immutable  Collections

Mutable  Collections

Simpler  is  safer  Indexing  by  first  character

Java List<String> keywords = Arrays.asList("Apple", "Banana", "Beer"); Map<Character, List<String>> result = new HashMap<Character, List<String>>(); for(String k : keywords) { char firstChar = k.charAt(1); if(!result.containsKey(firstChar)) { result.put(firstChar, new ArrayList<String>());

} result.get(firstChar).add(k); } for (List<String> list : result.values()) { Collections.sort(list); } Scala val keywords = List("Apple", "Banana", "Beer”) val result = keywords.sorted.groupBy(_.head)

Oops,  here’s  a  bug.  Anyone  noticed?

Another  example !Java!public List<String> empNames(ArrayList<Employee> employees) {!

!ArrayList<String> result = new ArrayList<String>();!!for (Employee emp: employees) {!! !result.add(emp.getName());!!}!!return result;!

}!!!!Scala!def empNames(employees: List[Employee]) = employees map getName!

Map  combinator  paVern !Java!public List<String> empNames(ArrayList<Employee> employees) {!

!ArrayList<String> res = new ArrayList<String>();!!for (Employee emp: employees) {!! !res.add(emp.getName());!!}!!return res;!

}!!!!Scala!def empNames(employees: List[Employee]) = employees map getName!

Really  need  to  encapsulate??

Another  example !Java!public static int factorial(int n) {! int res = 1;! for (int i = 1; i <= n; i++) {! res *= i;! }! return res;!

}!!!

Scala!def factorial(n: Int) = (1 to n).reduce(_*_)!

Reduce  combinator  paVern !Java!public static int factorial(int n) {! int res = 1;! for (int i = 1; i <= n; i++) {! res *= i;! }! return res;!

}!!!

Scala!def factorial(n: Int) = (1 to n).reduce(_ * _)!

Combinatorics o  (1 to 5) combinations 2

List(Vector(1, 2), Vector(1, 3), Vector(1, 4), Vector(1, 5), Vector(2, 3), Vector(2, 4), Vector(2, 5), Vector(3, 4), Vector(3, 5), Vector(4, 5))

o  (1 to 5).permutations "George W. Bush".split(" ").permutations List(Array(George, W., Bush), Array(George, Bush, W.), Array(W., George, Bush), Array(W., Bush, George), Array(Bush, George, W.), Array(Bush, W., George))

o  "George W. Bush".split(" ").toSet.subsets List(Set(), Set(George), Set(W.), Set(Bush), Set(George, W.), Set(George, Bush), Set(W., Bush), Set(George, W., Bush))

Collection  framework ++ ++: +: /: /:\ !:+ :: ::: :\ addString !aggregate andThen apply applyOrElse asInstanceOf !canEqual collect collectFirst combinations companion !compose contains containsSlice copyToArray copyToBuffer !

corresponds count diff distinct drop !dropRight dropWhile endsWith exists filter !filterNot find flatMap flatten fold !foldLeft foldRight forall foreach genericBuilder !groupBy grouped hasDefiniteSize head headOption !indexOf indexOfSlice indexWhere indices init !inits intersect isDefinedAt isEmpty isInstanceOf !isTraversableAgain iterator last lastIndexOf lastIndexOfSlice !

lastIndexWhere lastOption length lengthCompare lift !map mapConserve max maxBy min !minBy mkString nonEmpty orElse padTo !par partition patch permutations prefixLength !product productArity productElement productIterator productPrefix !reduce reduceLeft reduceLeftOption reduceOption reduceRight !reduceRightOption repr reverse reverseIterator reverseMap !reverse_::: runWith sameElements scan scanLeft !

scanRight segmentLength seq size slice !sliding sortBy sortWith sorted span !splitAt startsWith stringPrefix sum tail !tails take takeRight takeWhile to !toArray toBuffer toIndexedSeq toIterable toIterator !toList toMap toSeq toSet toStream !toString toTraversable toVector transpose union !unzip unzip3 updated view withFilter !zip zipAll zipWithIndex !

Placeholder  syntax  for  anonymous  functions

_ + 1 x => x + 1 _ * _ (x1, x2) => x1 * x2

(_: Int) * 2 (x: Int) => x * 2 if (_) x else y z => if (z) x else y

_.map(f) x => x.map(f)

_.map(_ + 1) x => x.map(y => y + 1)

Regular Placeholder

Distributed  &  multicore  computing

•  Parallel collections (myList.par.sum) •  Futures & promises •  Actors •  STM – makes Java heap ACID compatible

•  Hadoop •  Spark & Storm •  Plain old Java threads •  Scalding – Twitter’s Scala based MapReduce

implementation

Core

3rd  P

arty

Calling  Hadoop  from  Scala

Java public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {

String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } }

Scala def map(key: LongWritable, value: Text, output: OutputCollector[Text, IntWritable], reporter: Reporter) = value split " " foreach (output collect (_, one))

Futures

val from1 = future {conn1.fetch} val from2 = future {conn2.fetch}   from1 either from2 onSuccess println

Parallel  collections List(1, 2, 3, 4, 5).par filter (_ % 2 == 0) // ParVector(2, 4)

Spark val file = spark textFile "hdfs://...” !file.flatMap(_ split " ") .map(word => (word, 1)) .reduceByKey(_ + _)!!

Tail  recursion  optimization

Recursion in Java has an Achilles’ heel: the call stack def factorial(n: Int, res: Long = 1): Long = if(n == 0) res else factorial(n - 1, (res * n))

Safer  string  composition •  Standard string composition:

“I’m “ + name + “, “ + age + “ years old”

•  Formatted: “I’m %s, %d years old”.format(name, age) java.util.IllegalFormatConversionException

•  Interpolated: s“I’m $name, $age years old”

Create  your  own  interpolations

xml”””<body>

<a href = “http://…”> $text</a> </body> ”””

Expression  simplification  –  Java public Expr simplify(Expr expr) { if (expr instanceof UnOp) { UnOp unOp = (UnOp) expr; if (unOp.getOperator().equals("-")) { if (unOp.getArg() instanceof UnOp) { UnOp arg = (UnOp) unOp.getArg(); if (arg.getOperator().equals("-")) return arg.getArg(); } } } if (expr instanceof BinOp) { BinOp binOp = (BinOp) expr; if (binOp.getRight() instanceof Number) { Number arg = (Number) binOp.getRight(); if (binOp.getOperator().equals("+") && arg.getNum() == 0) return binOp.getLeft(); } } return expr; }

PaVern  matching  Expression  simplification  -­‐‑  Scala

def simplify(expr: Expr): Expr = expr match {! case UnOp("-", UnOp("-", e)) => simplify(e)! case BinOp("+", e, Number(0)) => simplify(e)! case _ => expr!}!

Value  classes Extensibility minus boxing overhead!!class Meter(val v: Double) extends AnyVal {! def plus (that: Meter) = new Meter(v + that.v) !}!!

Compiler generates!

object Meter {! def plus(this: Meter, that: Meter) = new Meter(v + that.v)!}!

Compile-­‐‑time  Meta-­‐‑programming

•  Language virtualization (overloading/overriding semantics of the original programming language to enable deep embedding of DSLs),

•  Program reification (providing programs with means to inspect their own code),

•  Self-optimization (self-application of domain-specific optimizations based on program reification),

•  Algorithmic program construction (generation of code that is tedious to write with the abstractions supported by a programming language).

Meta  programming Language Operate  on Type  

safe Expressiveness

Runtime  Performance  overhead

Textual  preprocessors

C,  Scala Text ✗ Low No

Template  systems

C++ Text ✗ Low No

Compile-­‐‑time  meta-­‐‑programming  

Haskell,  Scala  2.10,  Nemerle

Abstract  (typed)  Syntax  trees

✔ High

No

Byte  code  manipulation

Java,  Scala byte-­‐‑code ✗ Medium No

“Dynamic”  reflection

Ruby,  Scala objects ✗ High Yes

Run-­‐‑time  reflection

Java,  Scala objects ✗ High Yes

Scala  macros •  Full blown compile time meta-

programming •  Access to the compiler API •  Written in Scala •  Hygienic •  Type-safe

Macros  –  safe  printf printf(format: String, params: Any*)!printf(“value is %d”, “boom”)!// runtime exception!java.util.IllegalFormatConversionException: d != java.lang.String!!!

Macro implementation!def printf(format: String, params: Any*): Unit = macro printf_impl!def printf_impl(c: Context)(format: c.Expr[String], params: c.Expr[Any]*): c.Expr[Unit] = ...!

Efficient  Assert assert(2 + 2 == 4, "weird arithmetic") // 2 + 2 == 4 will be evaluated only if assertions were enabled

Macros  –  units  of  measure val gravityOnEarth = u(9.81, "m/s^2")!

!

val heightOfMyOfficeWindow = u(3.5, "m")!

!

val speedOfImpact = sqrt(2.0 * gravityOnEarth * heightOfMyOfficeWindow)!

!

val monthsInAYear = 10b12!

hVp://scalamacros.org/usecases/units-­‐‑of-­‐‑measure.html

Macros  –  type  providers

type MySqlDb(connString: String) = macro ...!

type MyDb = Base with MySqlDb("Server=127.0.0.1;Database=Foo;”)!

val products = new MyDb().products!

products.filter(p => p.name.startsWith("foo"))! !

!!

!

!

http://scalamacros.org/usecases/type-providers.html!

Inspired by F# type providers!

Type  macros  Injecting  async  methods

class D extends Lifter {!!def x = 2!!// def asyncX = future { 2 }!

}!!val d = new D!d.asyncX onComplete {!

!case Success(x) => println(x)!!case Failure(_) => println("failed")!

}!

 hVp://scalamacros.org/talks/2012-­‐‑12-­‐‑18-­‐‑MacroParadise.pdf

More  uses  of  macros

•  Ad-hoc code style and code-smell detector •  Advanced domain-specific languages •  Logging with minimal overhead •  Pre-compiled SQL queries •  GPU optimized code (see Scalaxy, ScalaCL) •  Macro annotations

o  @Cached

Macro  annotations class atomic extends MacroAnnotation { def complete(defn: _) = macro(“backing field”) def typeCheck(defn: _) = macro("return defn itself”) } @atomic var fld: Int

GPU  compilation •  Enablers: Immutability, Macros •  ScalaCL Collections

OpenCL-backed collections that look and behave like standard Scala collections (work in progress).

•  ScalaCL Compiler Plugin optimizes Scala programs at compile-time, transforming regular Scala loops into faster code and transforming Scala functions given to ScalaCL Collections into OpenCL kernels.

An  extremely  hack-­‐‑able  compiler > scala -Xshow-phases! phase name id description! ---------- -- -----------! parser 1 parse source into ASTs, perform simple desugaring! namer 2 resolve names, attach symbols to named trees!packageobjects 3 load package objects! typer 4 the meat and potatoes: type the trees! patmat 5 translate match expressions!superaccessors 6 add super accessors in traits and nested classes! extmethods 7 add extension methods for inline classes! pickler 8 serialize symbol tables! refchecks 9 reference/override checking, translate nested objects! selectiveanf 10 ANF pre-transform for @cps! selectivecps 11 @cps-driven transform of selectiveanf assignments! uncurry 12 uncurry, translate function values to anonymous classes! tailcalls 13 replace tail calls by jumps! specialize 14 @specialized-driven class and method specialization! explicitouter 15 this refs to outer pointers, translate patterns! erasure 16 erase types, add interfaces for traits! posterasure 17 clean up erased inline classes! lazyvals 18 allocate bitmaps, translate lazy vals into lazified defs! lambdalift 19 move nested functions to top level! constructors 20 move field definitions into constructors! flatten 21 eliminate inner classes! mixin 22 mixin composition! cleanup 23 platform-specific cleanups, generate reflective calls! icode 24 generate portable intermediate code! inliner 25 optimization: do inlining!inlinehandlers 26 optimization: inline exception handlers! closelim 27 optimization: eliminate uncalled closures! dce 28 optimization: eliminate dead code! jvm 29 generate JVM bytecode! terminal 30 The last phase in the compiler chain!

Type-­‐‑safe  failures try { 2 / x } catch { case e:Throwable => 0} in 2.10: Try (2 / 0) // returns Failure Try (2 / 1) // returns Success Try (2 / x) getOrElse (0) List(0,1,0,2).map (x=> Try(2 / x)) filter(_.isSuccess) // List(Success(2), Success(1))

Implicit  parameters implicit val db = getDB!def store(employee: Employee)(implicit db: DB)!!store(emp1oyee1) //db is passed implicitly!

Implicit  methods implicit def toEuro(x: Currency): Euro = …!!def transfer(amount: Euro) {! println(”transferred” + amount)!}!!// dollar is automatically converter to euro!transfer(Dollar(3.0))!transferred 2.25 Euros!

Implicit  classes !implicit class StringExtensions(s: String) {! def words = s split " "!}!!!“hello world”.words // Array(hello, world)!

Environments

ublime   -­‐‑  Through  Ensime

-­‐‑  Formally  supported  by  TypeSafe

-­‐‑  Currently  the  most  mature

Scala  REPL

DSLs  (accounting) Rules to calculate an employee's paycheck:! employee's gross salary for 2 weeks! minus deductions for! federalIncomeTax, which is 25% of gross! stateIncomeTax, which is 5% of gross! insurancePremiums, which are 500 in gross’s currency! retirementFundContributions are 10% of gross!!

hVp://ofps.oreilly.com/titles/9780596155957/DomainSpecificLanguages.html

DSLs  (JSON) ("person" ->! ("name" -> "Joe") ~! ("age" -> 35) ~! ("spouse" ->! ("person" ->! ("name" -> "Marilyn") ~! ("age" -> 33)! )! )!)!

DSL  (XPath  /  json)

val titles = xml \ "channel" \ "item" \ "title“ val titles2 = json \ "feed" \ "entry" \ "title" \@ "$t"

github.com/razie/snakked

DSLs  testing  frameworks

val emptyStack = new Stack[String] evaluating { emptyStack.pop() } should

produce [NoSuchElementException]

Full  access  to  the  compiler import tools.reflect.ToolBox!import reflect.runtime.{currentMirror => cm}!!val tb = cm.mkToolBox()!!scala> val tree = tb parse "(5 + 9) * 3"!tree: tb.u.Tree = 5.$plus(9).$times(3)!!scala> tb eval tree!res1: Any = 42!

Reflection

val take = typeOf[List[_]].member(TermName("take")).asMethod!!reflect(List(1,3,2,4)).reflectMethod(take)(2) // List(1,2)!

Testing •  ScalaCheck

o  Auto-generation of inputs and mocks. o  Composable check units o  Inspired by Haskell QuickCheck

•  ScalaTest o  TDD, BDD, FeatureSpec o  Selenium DSL

•  Specs2 o  TDD, BDD, Datatables support o  Integration with Mockito, EasyMock and JMock o  "hello world" must be matching("h.* w.*")

•  JUnit

ScalaCheck  (inspired  by  Haskell  QuickCheck)

scala> val sqrt = forAll {(n: Int) => math.sqrt(n*n) == n } scala> propSqrt.check ! Falsified after 1 passed tests: > -1

BDD  with  ScalaTest describe("A Stack") { it("should throw NoSuchElementException if an empty stack is popped") in { val emptyStack = new Stack[String] evaluating { emptyStack.pop() } should produce [NoSuchElementException] } it("should pop values in last-in-first-out order") { … } }

How  to  learn

•  twitter.github.com/scala_school •  Effective Scala •  Coursera Scala course (50K students last year) •  Simply scala (interactive learning) •  Programming in scala book •  Stackoverflow

Contributing  docs.scala-­‐‑lang.org/sips

Contributing  github.com/scala

Things  to  be  aware  of ✗  Scala Compiler has to do much more => slower

✗  Tooling is not perfect (debugging, synthetic frames, macros support) Getting better every day.

✗  Language is rapidly evolving. Deprecations should be expected.