scala at netflix
DESCRIPTION
My talk at Scala Bay Meetup at Netflix about Powering the Partner APIs with Scalatra and Netflix OSS. This talk was delivered on September 9th 2013, at 8 PM at Netflix, Los Gatos.TRANSCRIPT
![Page 2: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/2.jpg)
Agenda
Background
Netflix OSS Components
Development
Deployment/Delivery
Open Floor
![Page 3: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/3.jpg)
Partner Product Engineering
Smart devices
+
Certification
=
Lots of Device Metadata!
![Page 4: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/4.jpg)
Model
Firmware
Screen Resolution
Subtitle Support
3D
DRM
Remote Control
Netflix SDK
…
![Page 5: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/5.jpg)
Architecture
Cassandra
HTTP Layer, and Manager Layer EVCache
Crowd/SSO
RDS
Astyanax
Netflix OSS Cloud Components
![Page 6: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/6.jpg)
Netflix OSS Components
https://github.com/netflix
http://techblog.netflix.com
![Page 7: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/7.jpg)
Simian army
![Page 8: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/8.jpg)
![Page 9: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/9.jpg)
Asgard
![Page 10: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/10.jpg)
Eureka
![Page 11: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/11.jpg)
Karyon
![Page 12: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/12.jpg)
Astyanax
https://github.com/Netflix/astyanax
Astyanax is a Java Client for Cassandra
![Page 13: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/13.jpg)
EVCache
![Page 14: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/14.jpg)
Development
We to code.
![Page 15: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/15.jpg)
Test first
We write a ton of tests.
![Page 16: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/16.jpg)
Test coverage is a measure of confidence.
![Page 17: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/17.jpg)
ScalaTest
simple
intuitive
english-like
rich
promotes BDD
![Page 18: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/18.jpg)
Test first
Manager Layer
No HTTP requests, simple method invocation
![Page 19: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/19.jpg)
/**Lets try to build a login endpoint. It should support a method calledlogin(user:String,pass:String) that returns an Option[String].
*/
class LoginManagerSpec extends FlatSpec with ShouldMatchers {
}
![Page 20: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/20.jpg)
/**Lets try to build a login endpoint. It should support a method calledlogin(user:String,pass:String) that returns an Option[String].
*/
class LoginManagerSpec extends FlatSpec with ShouldMatchers {
it should " Be able to login a valid user and get a token " in {fail()
}
}
![Page 21: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/21.jpg)
/**Lets try to build a login endpoint. It should support a method calledlogin(user:String,pass:String) that returns an Option[String].
*/
class LoginManagerSpec extends FlatSpec with ShouldMatchers {
it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None }}
![Page 22: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/22.jpg)
/**Lets try to build a login endpoint. It should support a method calledlogin(user:String,pass:String) that returns an Option[String].
*/
class LoginManagerSpec extends FlatSpec with ShouldMatchers {
it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None }
it should " Fail to login an invalid user " in {fail
}
}
![Page 23: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/23.jpg)
/**Lets try to build a login endpoint. It should support a method calledlogin(user:String,pass:String) that returns an Option[String].
*/
class LoginManagerSpec extends FlatSpec with ShouldMatchers {
it should " Be able to login a valid user and get a token " in { val token = LoginManager.login("someuser", "somepassword") token should not be None }
it should " Fail to login an invalid user " in { val token = LoginManager.login("fail", "fail") token should be (None) }
}
![Page 24: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/24.jpg)
HTTP, or the Scalatra Layer
![Page 25: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/25.jpg)
Scalatra
very simple, Sinatra-inspired framework
routes defined with code
json4s
Swagger
ScalatraSpec
![Page 26: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/26.jpg)
A Simple API in Scalatra
class LoginService extends ScalatraServlet
![Page 27: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/27.jpg)
A Simple API in Scalatra
class LoginService extends ScalatraServlet {
post("/") { }}
![Page 28: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/28.jpg)
A Simple API in Scalatra
class LoginService extends ScalatraServlet {
before() { contentType = formats("json") }
post("/") { val token = LoginManager.login(params("user"), params("password"))
token match{case None => Unauthorized(Map("message"->"Login failed"))case Some(x) => Ok(Map("token"->x))
} }}
![Page 29: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/29.jpg)
Putting it all together..
class ScalatraBootstrap extends LifeCycle {
override def init(context: ServletContext) {context.mount(new LoginService, "/login/*")
}}
<listener> <listener-class>org.scalatra.servlet.ScalatraListener</listener-class> </listener>
![Page 30: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/30.jpg)
$ curl http://localhost:8080/login --form "user=foo&password=bar"
{"token":"02055794-1928-11e3-9a3b-f23c91aec05e"}
![Page 31: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/31.jpg)
ScalatraSpec
traits to take ScalaTest to the next level
support for all HTTP methods
helpers for JSON parsing
plenty of wrappers (body, headers..)
![Page 32: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/32.jpg)
class LoginServiceSpec extends ScalatraFlatSpec {
}
![Page 33: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/33.jpg)
class LoginServiceSpec extends ScalatraFlatSpec {
addServlet(classOf[LoginService], "/*")
}
![Page 34: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/34.jpg)
class LoginServiceSpec extends ScalatraFlatSpec {
addServlet(classOf[LoginService], "/*")
it should "log in valid users" in { post("/", body = """user=gooduser&password=goodpassword""") { status should equal(200) body should include "token" } }}
![Page 35: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/35.jpg)
class LoginServiceSpec extends ScalatraFlatSpec {
addServlet(classOf[LoginService], "/*")
it should "log in valid users" in { post("/", body = """user=gooduser&password=goodpassword""") { status should equal(200) body should include "token" } }
it should "not allow invalid users to log in" in { post("/", body = """user=baduser&password=badpassword""") { status should equal(401) body should include "message" } }}
![Page 36: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/36.jpg)
APIs Best Practices
Use Proper HTTP Response Codes
Set Proper HTTP Headers
Break up your data into groups
![Page 37: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/37.jpg)
Pop Quiz!
Lets do some HTTP response codes..
![Page 38: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/38.jpg)
Pop Quiz!
What is the response code for an async operation?
![Page 39: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/39.jpg)
Pop Quiz!
…forbidden?
![Page 40: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/40.jpg)
Pop Quiz!
…a delete?
![Page 41: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/41.jpg)
Git Workflow
work on the dev branch
write tests
leave the rest to Jenkins
![Page 42: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/42.jpg)
Git Workflow
$ git status
# On branch dev# Changes not staged for commit:# (use "git add <file>..." to update what will be committed)# (use "git checkout -- <file>..." to discard changes in working directory)## modified: src/main/scala/com/netflix/nrdportal/http/DpiService.scala# modified: src/test/scala/com/netflix/nrdportal/http/DpiServiceSpec.scala
![Page 43: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/43.jpg)
Automated Code Pushes
Push to dev
Jenkins runs dev build,
tests, merges to
master
Jenkins runs master build,
makes an RPM
Aminator bakes an
AMI from the RPM
asgard deploys the
AMI in staging cloud
![Page 44: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/44.jpg)
Scala Best Practices
Using Options
Using Try[A] vs. Exceptions
Wrappers
Control Abstractions
![Page 45: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/45.jpg)
The dreaded null
public String willReturnNullForOdds(int x){ if(x%2==0) return "Even"; else return null;}
![Page 46: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/46.jpg)
Using Options
def willReturnNullForOdds(x: Int): Option[String] = { if (x % 2 == 0) Some("Even") else None }
![Page 47: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/47.jpg)
Using Options as Wrappers
…where you have to call code that can return null
def returnNull(x:Int) = if(x%2 == 0) "Even" else null
scala> Option(returnNull(3)) res01: Option[String] = None
scala> Option(returnNull(2)) res02: Option[String] = Some(Even)
![Page 48: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/48.jpg)
Exceptions?
public String willThrowExceptionForOdds(int x){if(x%2==0) return "Even";else throw new IllegalArgumentException("Odd Number!");
}
![Page 49: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/49.jpg)
Using Try[A]
def someFunction(x: Int): Try[String] = { if (x % 2 == 0) Success("Even") else Failure(new IllegalArgumentException("Odd number!")) }
![Page 50: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/50.jpg)
Control Abstractions
def withAuthenticatedUser(f: (String) => ActionResult) = { getUserNameFromCookie match { case Some(userName) => f(userName) case None => Unauthorized("You are not logged in!") } }
def printCurrentUserName = { withAuthenticatedUser { userName => Ok(s"Your username is ${userName}") } }
![Page 51: Scala at Netflix](https://reader033.vdocuments.mx/reader033/viewer/2022061613/554f90c9b4c905d25b8b51c7/html5/thumbnails/51.jpg)
Finally…
Avoid writing complex code at all costs – there are better ways to prove your awesomeness!