Go90                          

Vidhya Narayanan, Director, Verizon Christopher Webster, Associate Fellow, Verizon

October 2015

A Look inside go90 services architecture

Micro  service  is  a…  

a.  new  taxi  hailing  app  developed  by  a  middle  schooler  in  India  

b.  catering  service  run  by  bacteria    

c.  microwave-­‐as-­‐a-­‐service  from  Verizon  

d.  Complex  applica@on  broken  down  into  separate  services  

©2015  Apigee  Corp.  All  Rights  Reserved.    

The  views  expressed  in  this  presenta@on  are  those  of  the  presenter,  and  not  

necessarily  those  of  Apigee  Corpora@on  or  the  presenter’s  employer.  

3

go90.com, #go90 go90.com, #go90

Go90  —  from  30,000  P  

•  100+  micro  services/systems    

•  10+  deployments  per  week  

•  Mul@ple  clients,  large  target  user  base  

•  High  reliability/availability,  low  latency,  superior  user  experience  

 

In  the  beginning  

Goals  

•  Must  scale  and  be  highly  available  •  Support  mul@ple  video  products  •  Meet  security  requirements  •  Need  to  use  exis@ng  services  

 

Solu@on  

•  Services  with  contained  domain  logic  •  Resource  centric  APIs  that  expose  data  model  •  Common  libraries  •  Graceful  degrada@on  •  Security  readiness  

Discovery  STB  

Content  Ingest  

CDN  

Accounts  

Log  Upload  

Search  

Profile  Image  

Discovery  

   

Event  Q  

DRM  

Analy@cs  

Favorites  

Cert  Prov  

AWS  HSM  

Ads  

Offers  and  Packages  

DAI  backend  

STB  

Content  Ingest  

CDN  

Reco  

Discovery  

Accounts  

Favorites  

Search  

DRM  

Profile  

Rights  

SU  

and…a  100  others  

Profiles      

Profiles  Q  

Challenges  

•  Performance  •  Uniformity    •  Security  

Evolu7on  

Added  Goals  

•  Reduce  development  @me  for  new  client  placorms  

•  Improve  release  velocity    •  SLA  for  call  latency  •  Keep  security  simple  

   

Evolu@on  1  

•  Uniform  API  

Governor  

STB   Maestro  

Content  Ingest  

CDN  

Governor  

Accounts  

Log  Upload  

Search  

Profile  Image  

Discovery  

   

Event  Q  

DRM  

Analy@cs  

Favorites  

Cert  Prov  

AWS  HSM  

Ads  

Offers  and  Packages  

DAI  backend  

STB   Maestro  

Content  Ingest  

CDN  

Reco  

Discovery  

Accounts  

Favorites  

Search  

DRM  

Profile  

Rights  

SU  

and…a  100  others  

Profiles      

Profiles  Q  

Challenges  

•  Monolith  •  Itera@on  with  fast  changing  requirements  

Evolu@on  2  

•  Microservice  enhancements  •  More  preprocessing  •  Remove  Logic  from  edge  services  

 

Scale  

Our  Approach  

Model  Predicted  Load  

Develop  and  

Execute  Load  Test  

Op@mize  /  Scale  for  Load  Test  

Validate  Model  against  

known  data  

Our  Approach  

•  Benchmark  individual  APIs  •  Develop  model    •  Execute  cycle  on  system    •  Verify  model  •  Launch  •  Use  produc@on  traffic  logs  •  Inject  failure  during  load  tests  

Benchmark  individual  APIs  

•  Needed  20x  improvement  •  Stress  APIs  and  get  to  a  no  crash  milestone  

Scenario  Runs  

•  First  run    •  Op@mized  app  call  pagern  •  Model  itera@on  using  earlier  results  and  expected  users  projec@ons  

•  Execute  app  scenario  (on  en@re  system)  with  larger  number  of  users,  but  s@ll  10x  less  than  target  

Validate  Scenarios  

•  Increased  scenario  to  target  concurrent  users  •  Added  think  @me  to  scenario  •  Ran  scenario  with  target  users  •  Scenario  running  with  load  <  .1  •  Chaos  Monkey  

Summary  

•  Measure  first,  op@mize  later  •  Start  simple  •  Dev  v.  DevOps    •  Minimize  infrastructure  debt