Making better news with Spark
Phil Wills@philwills
What problems were we trying to solve?
What issues have we encountered?
What’s worked well?
What problems were we trying to solve?
What issues have we encountered?
ExecutorLostFailure (executor lost)
object Foo { def bar(i: Int) = i + 1 def go() = sc.parallelize(List(1,2,3)) .map(bar(_)) .toArray()
}
Foo.go()
object Foo2 { val bar = (i: Int) => i + 1 def go() = sc.parallelize(List(1,2,3)) .map(bar) .toArray()
}
Foo2.go()
Array[Int] = Array(2, 3, 4)
© Copyright Albert Bridge
--num-executors--executor-cores--executor-memory--driver-cores--driver-memory